2024-02-14T17:05:28.151 INFO:root:teuthology version: 0.0.1.dev258+g9e44c16 2024-02-14T17:05:28.152 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762 --verbose --name gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} -- /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762/orig.config.yaml 2024-02-14T17:05:28.198 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-14T17:05:28.290 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762 branch: wip-guits-testing-2024-02-14-1033 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: gabrioux@ibm.com first_in_suite: false job_id: '7559762' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-02-14-1033 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 workunit: branch: wip-guits-testing-2024-02-14-1033 sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 4718 sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-02-14-1033 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8727ce939fc044a98f215e29696bb1433f5758d1/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlgQNIOt312vnrlk/zAL3nwp0I8wQzfpfijrVc76T3z87q3Ip+MOw2OXO5VHKIV67nkUS2ecXCUHf+mUO77nHM= smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJsK8rDeLbpPpgvN/M2P+/nrEkMeAEFZvPf+7wnuIgV9YVJjjxcH1OHUvmmwLUjAzWuoKlaV/XCUjGvNf+2djj8= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9e44c16382dfc4c612f7437cbb344c76223137cc timestamp: 2024-02-14_13:54:10 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-14T17:05:28.290 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8727ce939fc044a98f215e29696bb1433f5758d1/qa; will attempt to use it 2024-02-14T17:05:28.291 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8727ce939fc044a98f215e29696bb1433f5758d1/qa/tasks 2024-02-14T17:05:28.291 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-14T17:05:28.293 INFO:teuthology.task.internal:Checking packages... 2024-02-14T17:05:28.323 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8727ce939fc044a98f215e29696bb1433f5758d1' 2024-02-14T17:05:28.323 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-14T17:05:28.323 INFO:teuthology.packaging:ref: None 2024-02-14T17:05:28.323 INFO:teuthology.packaging:tag: None 2024-02-14T17:05:28.323 INFO:teuthology.packaging:branch: wip-guits-testing-2024-02-14-1033 2024-02-14T17:05:28.323 INFO:teuthology.packaging:sha1: 8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:05:28.323 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-02-14-1033 2024-02-14T17:05:28.548 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1430.g8727ce93 2024-02-14T17:05:28.549 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-14T17:05:28.556 INFO:teuthology.task.internal:no buildpackages task found 2024-02-14T17:05:28.556 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-14T17:05:28.600 INFO:teuthology.task.internal:Saving configuration 2024-02-14T17:05:28.618 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-14T17:05:28.624 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-14T17:05:28.645 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 16:59:55.993724', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlgQNIOt312vnrlk/zAL3nwp0I8wQzfpfijrVc76T3z87q3Ip+MOw2OXO5VHKIV67nkUS2ecXCUHf+mUO77nHM='} 2024-02-14T17:05:28.667 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 16:59:55.991912', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJsK8rDeLbpPpgvN/M2P+/nrEkMeAEFZvPf+7wnuIgV9YVJjjxcH1OHUvmmwLUjAzWuoKlaV/XCUjGvNf+2djj8='} 2024-02-14T17:05:28.667 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-14T17:05:28.673 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-02-14T17:05:28.673 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-14T17:05:28.673 INFO:teuthology.run_tasks:Running task console_log... 2024-02-14T17:05:28.763 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f58491a50d0>, signals=[15]) 2024-02-14T17:05:28.764 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-14T17:05:28.770 INFO:teuthology.task.internal:Opening connections... 2024-02-14T17:05:28.770 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-02-14T17:05:28.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:05:28.880 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-02-14T17:05:28.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:05:28.974 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-14T17:05:28.981 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-02-14T17:05:29.000 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-02-14T17:05:29.000 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-02-14T17:05:29.058 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-02-14T17:05:29.058 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-02-14T17:05:29.058 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-02-14T17:05:29.058 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-02-14T17:05:29.058 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-14T17:05:29.059 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-14T17:05:29.060 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-02-14T17:05:29.088 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-02-14T17:05:29.105 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-02-14T17:05:29.105 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-14T17:05:29.162 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-02-14T17:05:29.163 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-02-14T17:05:29.163 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-14T17:05:29.163 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-02-14T17:05:29.163 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-14T17:05:29.163 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-14T17:05:29.163 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-14T17:05:29.163 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-02-14T17:05:29.187 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-14T17:05:29.195 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-14T17:05:29.217 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-14T17:05:29.217 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-02-14T17:05:29.220 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-02-14T17:05:29.235 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-14T17:05:29.241 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-14T17:05:29.241 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-02-14T17:05:29.278 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-02-14T17:05:29.305 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-14T17:05:29.358 INFO:teuthology.run_tasks:Running task kernel... 2024-02-14T17:05:29.372 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-14T17:05:29.372 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-14T17:05:29.372 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-14T17:05:29.373 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-14T17:05:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:05:29.388 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-02-14T17:05:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:05:29.390 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-02-14T17:05:29.444 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-361.el9.x86_64 2024-02-14T17:05:29.444 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.14.0-361.el9.x86_64 2024-02-14T17:05:29.444 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-02-14T17:05:29.448 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-361.el9.x86_64 2024-02-14T17:05:29.448 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-361.el9.x86_64 2024-02-14T17:05:29.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-02-14T17:05:31.484 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-02-14T17:05:31.585 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-14T17:05:33.310 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-02-14T17:05:33.325 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-02-14T17:05:34.858 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-02-14T17:05:35.592 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-02-14T17:05:37.459 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-14T17:05:42.493 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.1 MB 00:01 2024-02-14T17:05:44.052 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-02-14T17:05:45.455 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-02-14T17:05:50.570 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-02-14T17:05:52.918 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-14T17:05:52.919 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-14T17:05:53.007 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-420.el9 CentOS-BaseOS 6.0 M 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-420.el9 CentOS-BaseOS 21 M 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-420.el9 CentOS-BaseOS 39 M 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-420.el9 CentOS-BaseOS 33 M 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:05:53.013 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-02-14T17:05:53.014 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-02-14T17:05:53.014 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-02-14T17:05:53.014 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:05:53.014 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 99 M 2024-02-14T17:05:53.014 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 122 M 2024-02-14T17:05:53.014 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-02-14T17:05:53.646 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-420.el9.x86_64.rpm 9.5 MB/s | 6.0 MB 00:00 2024-02-14T17:05:54.188 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-core-5.14.0-420.el9.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-02-14T17:05:54.905 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-modules-5.14.0-420.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-14T17:05:55.372 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-modules-core-5.14.0-420.el9.x86_6 19 MB/s | 33 MB 00:01 2024-02-14T17:05:55.375 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:05:55.375 INFO:teuthology.orchestra.run.smithi017.stdout:Total 42 MB/s | 99 MB 00:02 2024-02-14T17:05:55.376 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-02-14T17:05:55.802 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-02-14T17:05:55.802 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-02-14T17:05:55.892 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 1.0 MB/s | 19 MB 00:19 2024-02-14T17:05:56.893 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-02-14T17:05:56.894 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-02-14T17:05:59.427 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-02-14T17:05:59.866 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-02-14T17:06:00.105 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:06:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:06:01.327 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2024-02-14T17:06:03.159 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:06:03.358 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:06:03.762 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 20 MB 00:02 2024-02-14T17:06:08.452 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:06:08.546 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:08.668 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-02-14T17:06:10.975 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-14T17:06:10.975 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-14T17:06:11.064 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 5.14.0-420.el9 CentOS-BaseOS 6.0 M 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 5.14.0-420.el9 CentOS-BaseOS 21 M 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 5.14.0-420.el9 CentOS-BaseOS 39 M 2024-02-14T17:06:11.075 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core x86_64 5.14.0-420.el9 CentOS-BaseOS 33 M 2024-02-14T17:06:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:06:11.076 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-14T17:06:11.076 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-14T17:06:11.076 INFO:teuthology.orchestra.run.smithi111.stdout:Install 4 Packages 2024-02-14T17:06:11.076 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:06:11.076 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 99 M 2024-02-14T17:06:11.076 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 122 M 2024-02-14T17:06:11.077 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-14T17:06:11.478 INFO:teuthology.orchestra.run.smithi111.stdout:(1/4): kernel-5.14.0-420.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-02-14T17:06:12.103 INFO:teuthology.orchestra.run.smithi111.stdout:(2/4): kernel-core-5.14.0-420.el9.x86_64.rpm 20 MB/s | 21 MB 00:01 2024-02-14T17:06:12.778 INFO:teuthology.orchestra.run.smithi111.stdout:(3/4): kernel-modules-core-5.14.0-420.el9.x86_6 26 MB/s | 33 MB 00:01 2024-02-14T17:06:13.345 INFO:teuthology.orchestra.run.smithi111.stdout:(4/4): kernel-modules-5.14.0-420.el9.x86_64.rpm 17 MB/s | 39 MB 00:02 2024-02-14T17:06:13.348 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-14T17:06:13.348 INFO:teuthology.orchestra.run.smithi111.stdout:Total 44 MB/s | 99 MB 00:02 2024-02-14T17:06:13.349 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-14T17:06:13.658 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:13.714 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-14T17:06:13.714 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-14T17:06:14.729 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-14T17:06:14.730 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-14T17:06:17.117 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-14T17:06:17.768 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-core-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:06:17.905 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:06:20.685 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:06:20.898 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:06:25.594 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:06:25.710 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:30.448 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:44.290 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:44.290 INFO:teuthology.orchestra.run.smithi017.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-14T17:06:44.291 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:06:44.364 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:44.769 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:44.769 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:06:44.769 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:06:44.769 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:06:45.175 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-420.el9.x86_64 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-420.el9.x86_64 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-420.el9.x86_64 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-420.el9.x86_64 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:06:45.176 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-02-14T17:06:45.701 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:06:45.842 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:06:45.843 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:06:45.843 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-420.el9.x86_64 2024-02-14T17:06:45.843 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-14T17:06:45.843 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-14T17:06:45.843 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-14T17:06:45.843 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-02-14T17:06:45.844 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:06:45.956 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:06:45.956 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:06:45.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-02-14T17:06:45.999 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-02-14T17:06:45.999 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-02-14T17:06:45.999 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-02-14T17:06:45.999 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 68.el9 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8264254 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-14T17:06:46.000 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:06:46.001 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-02-14T17:06:46.002 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-14T17:06:46.002 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-420.el9.x86_64 2024-02-14T17:06:46.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-14T17:06:46.621 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-02-14T17:06:48.091 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-14T17:06:48.121 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-02-14T17:06:48.124 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-02-14T17:06:48.140 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.7DcNnpKPFf 2024-02-14T17:06:48.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7DcNnpKPFf 2024-02-14T17:06:48.213 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.7DcNnpKPFf 2024-02-14T17:06:48.357 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.7DcNnpKPFf is 6KB 2024-02-14T17:06:48.406 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.7DcNnpKPFf 2024-02-14T17:06:48.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-02-14T17:06:48.497 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-14T17:06:48.497 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-14T17:06:48.497 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-14T17:06:48.497 INFO:teuthology.orchestra.run.smithi017.stdout:15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64.conf 2024-02-14T17:06:48.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64 2024-02-14T17:06:48.655 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-02-14T17:06:59.720 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:06:59.721 INFO:teuthology.orchestra.run.smithi111.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-14T17:06:59.721 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:06:59.791 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:07:00.233 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:07:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-5.14.0-420.el9.x86_64 1/4 2024-02-14T17:07:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-5.14.0-420.el9.x86_64 2/4 2024-02-14T17:07:00.234 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-5.14.0-420.el9.x86_64 3/4 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-core-5.14.0-420.el9.x86_64 4/4 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-5.14.0-420.el9.x86_64 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core-5.14.0-420.el9.x86_64 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-5.14.0-420.el9.x86_64 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core-5.14.0-420.el9.x86_64 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:07:00.640 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-14T17:07:01.152 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:07:01.329 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:07:01.330 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:07:01.330 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-420.el9.x86_64 2024-02-14T17:07:01.330 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-14T17:07:01.330 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-14T17:07:01.330 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-14T17:07:01.330 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2024-02-14T17:07:01.330 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-02-14T17:07:01.437 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-420.el9.x86_64 2024-02-14T17:07:01.438 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-420.el9.x86_64 2024-02-14T17:07:01.438 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-02-14T17:07:01.471 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.06 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 68.el9 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Group : Unspecified 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 8264254 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-14T17:07:01.472 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : builder@centos.org 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:07:01.473 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-14T17:07:01.474 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-14T17:07:01.474 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-14T17:07:01.474 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-02-14T17:07:01.474 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:07:01.474 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-02-14T17:07:01.476 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-14T17:07:01.476 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 5.14.0-420.el9.x86_64 2024-02-14T17:07:01.476 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-14T17:07:02.121 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-02-14T17:07:03.528 INFO:teuthology.orchestra.run.smithi111.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-14T17:07:03.568 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-02-14T17:07:03.575 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-02-14T17:07:03.630 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.IdjJEGatB5 2024-02-14T17:07:03.630 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IdjJEGatB5 2024-02-14T17:07:03.699 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.IdjJEGatB5 2024-02-14T17:07:03.837 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.IdjJEGatB5 is 6KB 2024-02-14T17:07:03.885 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.IdjJEGatB5 2024-02-14T17:07:03.899 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-02-14T17:07:03.972 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-14T17:07:03.972 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-14T17:07:03.972 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-14T17:07:03.972 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64.conf 2024-02-14T17:07:03.974 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-420.el9.x86_64 2024-02-14T17:07:04.139 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-02-14T17:07:18.669 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-14T17:07:18.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-02-14T17:07:18.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:07:34.156 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-14T17:07:34.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:07:34.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:07:37.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-02-14T17:07:46.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-02-14T17:07:46.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:07:52.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-14T17:08:01.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:08:01.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:08:04.576 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-14T17:08:16.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:08:16.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:08:19.648 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-14T17:08:34.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:08:34.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:08:46.172 DEBUG:teuthology.orchestra.remote:timed out 2024-02-14T17:08:53.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-14T17:08:58.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-02-14T17:08:58.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:08:58.589 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-02-14T17:08:58.979 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-02-14T17:08:58.979 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-420.el9.x86_64"... 2024-02-14T17:08:58.979 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-02-14T17:08:58.994 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-420.el9.x86_64 2024-02-14T17:08:58.994 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-420.el9.x86_64 vs 5.14.0-420.el9.x86_64 2024-02-14T17:08:58.995 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-14T17:08:58.995 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-14T17:08:59.995 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-14T17:08:59.996 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-14T17:09:00.079 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-02-14T17:09:00.116 DEBUG:teuthology.parallel:result is None 2024-02-14T17:09:11.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:09:11.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:09:11.447 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-02-14T17:09:11.816 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:09:11.817 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-420.el9.x86_64"... 2024-02-14T17:09:11.817 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-02-14T17:09:11.831 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-420.el9.x86_64 2024-02-14T17:09:11.831 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-420.el9.x86_64 vs 5.14.0-420.el9.x86_64 2024-02-14T17:09:11.831 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-14T17:09:11.831 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-14T17:09:12.832 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-14T17:09:12.833 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-14T17:09:12.907 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-02-14T17:09:12.946 DEBUG:teuthology.parallel:result is None 2024-02-14T17:09:12.946 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-14T17:09:12.954 INFO:teuthology.task.internal:Creating test directory... 2024-02-14T17:09:12.954 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-14T17:09:12.957 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-14T17:09:12.974 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-14T17:09:13.000 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-14T17:09:13.033 INFO:teuthology.task.internal:Creating archive directory... 2024-02-14T17:09:13.034 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-14T17:09:13.036 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-14T17:09:13.072 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-14T17:09:13.079 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-14T17:09:13.079 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-14T17:09:13.100 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-14T17:09:13.133 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:09:13.139 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:09:13.147 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:09:13.152 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-14T17:09:13.154 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-14T17:09:13.161 INFO:teuthology.task.internal:Configuring sudo... 2024-02-14T17:09:13.161 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-14T17:09:13.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-14T17:09:13.221 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-14T17:09:13.234 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-14T17:09:13.234 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-14T17:09:13.262 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-14T17:09:13.277 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:09:13.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:09:13.407 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:09:13.489 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:09:13.534 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:09:13.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-14T17:09:13.603 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:09:13.644 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-14T17:09:13.699 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:09:13.784 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-14T17:09:13.826 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-14T17:09:13.826 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-14T17:09:13.897 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-02-14T17:09:13.899 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-02-14T17:09:13.947 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:09:13.987 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:09:14.500 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-14T17:09:14.508 INFO:teuthology.task.internal:Starting timer... 2024-02-14T17:09:14.508 INFO:teuthology.run_tasks:Running task pcp... 2024-02-14T17:09:14.553 INFO:teuthology.run_tasks:Running task selinux... 2024-02-14T17:09:14.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-02-14T17:09:14.680 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-02-14T17:09:14.682 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-02-14T17:09:14.732 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-02-14T17:09:14.734 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-14T17:09:14.734 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-02-14T17:09:14.758 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-02-14T17:09:14.758 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-02-14T17:09:14.780 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-02-14T17:09:14.780 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi111.front.sepia.ceph.com': 'permissive'} 2024-02-14T17:09:14.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:09:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:09:14.826 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:09:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:09:14.855 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-14T17:09:14.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-02-14T17:09:14.900 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-02-14T17:09:14.931 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-14T17:09:14.941 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-14T17:09:14.963 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-14T17:09:14.965 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi111.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-14T17:13:31.546 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi111.front.sepia.ceph.com')] 2024-02-14T17:13:31.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-02-14T17:13:31.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:13:31.645 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-02-14T17:13:31.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-02-14T17:13:31.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:13:31.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-14T17:13:31.819 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-02-14T17:13:31.896 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-14T17:13:31.896 INFO:teuthology.run_tasks:Running task clock... 2024-02-14T17:13:31.907 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-14T17:13:31.907 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-14T17:13:31.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:13:31.910 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-14T17:13:31.910 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:13:31.939 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-14T17:13:31.956 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-14T17:13:31.968 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-14T17:13:31.985 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-14T17:13:31.986 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-02-14T17:13:32.000 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-02-14T17:13:32.014 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-02-14T17:13:32.016 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-14T17:13:32.027 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-02-14T17:13:32.033 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-14T17:13:32.044 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-14T17:13:32.060 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-14T17:13:32.089 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:13:32.093 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:13:32.093 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-02-14T17:13:32.093 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.093 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.093 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.093 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.125 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:13:32.129 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:13:32.129 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-02-14T17:13:32.129 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.129 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.129 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.129 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:13:32.130 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-14T17:13:32.225 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '8727ce939fc044a98f215e29696bb1433f5758d1'} 2024-02-14T17:13:32.225 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-02-14T17:13:32.225 INFO:tasks.cephadm:Cluster fsid is 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:13:32.225 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-14T17:13:32.226 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.17', 'mon.c': '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]', 'mon.b': '172.21.15.111'} 2024-02-14T17:13:32.226 INFO:tasks.cephadm:First mon is mon.a on smithi017 2024-02-14T17:13:32.226 INFO:tasks.cephadm:First mgr is y 2024-02-14T17:13:32.226 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-14T17:13:32.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo hostname $(hostname -s) 2024-02-14T17:13:32.253 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-02-14T17:13:32.281 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-02-14T17:13:32.282 DEBUG:teuthology.orchestra.run.smithi017:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-14T17:13:32.563 INFO:teuthology.orchestra.run.smithi017.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 14 17:13 /home/ubuntu/cephtest/cephadm 2024-02-14T17:13:32.564 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-14T17:13:32.730 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Feb 14 17:13 /home/ubuntu/cephtest/cephadm 2024-02-14T17:13:32.731 DEBUG:teuthology.orchestra.run.smithi017:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-14T17:13:32.746 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-14T17:13:32.772 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-02-14T17:13:32.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-14T17:13:32.789 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-02-14T17:13:33.713 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-14T17:13:33.764 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-14T17:14:01.231 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:14:01.231 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-14T17:14:01.231 INFO:teuthology.orchestra.run.smithi017.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-14T17:14:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: "repo_digests": [ 2024-02-14T17:14:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-14T17:14:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-14T17:14:01.232 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-02-14T17:14:01.232 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:14:01.855 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-02-14T17:14:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-02-14T17:14:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-02-14T17:14:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-02-14T17:14:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-02-14T17:14:01.855 INFO:teuthology.orchestra.run.smithi111.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-02-14T17:14:01.856 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-02-14T17:14:01.856 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-02-14T17:14:01.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph 2024-02-14T17:14:01.908 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-02-14T17:14:01.939 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /etc/ceph 2024-02-14T17:14:01.979 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-02-14T17:14:02.006 INFO:tasks.cephadm:Writing seed config... 2024-02-14T17:14:02.007 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-14T17:14:02.007 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-14T17:14:02.007 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-14T17:14:02.007 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-02-14T17:14:02.007 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-14T17:14:02.008 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-14T17:14:02.008 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-14T17:14:02.008 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-14T17:14:02.008 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-14T17:14:02.008 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:14:02.008 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-14T17:14:02.036 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 612435b4-cb5c-11ee-95b9-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-14T17:14:02.036 DEBUG:teuthology.orchestra.run.smithi017:mon.a> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service 2024-02-14T17:14:02.080 DEBUG:teuthology.orchestra.run.smithi017:mgr.y> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service 2024-02-14T17:14:02.123 INFO:tasks.cephadm:Bootstrapping... 2024-02-14T17:14:02.123 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.17 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:14:02.327 INFO:teuthology.orchestra.run.smithi017.stderr:-------------------------------------------------------------------------------- 2024-02-14T17:14:02.327 INFO:teuthology.orchestra.run.smithi017.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '612435b4-cb5c-11ee-95b9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.17', '--skip-admin-label'] 2024-02-14T17:14:02.348 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.3-dev 2024-02-14T17:14:02.351 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying podman|docker is present... 2024-02-14T17:14:02.371 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.3-dev 2024-02-14T17:14:02.374 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying lvm2 is present... 2024-02-14T17:14:02.374 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying time synchronization is in place... 2024-02-14T17:14:02.381 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-14T17:14:02.387 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-02-14T17:14:02.393 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: enabled 2024-02-14T17:14:02.399 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: active 2024-02-14T17:14:02.399 INFO:teuthology.orchestra.run.smithi017.stderr:Unit chronyd.service is enabled and running 2024-02-14T17:14:02.399 INFO:teuthology.orchestra.run.smithi017.stderr:Repeating the final host check... 2024-02-14T17:14:02.421 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: 4.9.3-dev 2024-02-14T17:14:02.423 INFO:teuthology.orchestra.run.smithi017.stderr:podman (/bin/podman) version 4.9.3 is present 2024-02-14T17:14:02.423 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl is present 2024-02-14T17:14:02.423 INFO:teuthology.orchestra.run.smithi017.stderr:lvcreate is present 2024-02-14T17:14:02.429 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-14T17:14:02.478 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: enabled 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: active 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Unit chronyd.service is enabled and running 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Host looks OK 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Cluster fsid: 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Acquiring lock 140665196781472 on /run/cephadm/612435b4-cb5c-11ee-95b9-87774f69a715.lock 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Lock 140665196781472 acquired on /run/cephadm/612435b4-cb5c-11ee-95b9-87774f69a715.lock 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying IP 172.21.15.17 port 3300 ... 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying IP 172.21.15.17 port 6789 ... 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:Base mon IP is 172.21.15.17, final addrv is [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.17 metric 100 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.17 metric 100 2024-02-14T17:14:02.479 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe59:d992/64 scope link noprefixroute 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:Mon IP `172.21.15.17` is in CIDR network `172.21.0.0/20` 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-14T17:14:02.480 INFO:teuthology.orchestra.run.smithi017.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-02-14T17:14:02.563 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-02-14T17:14:03.086 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Getting image source signatures 2024-02-14T17:14:03.086 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-02-14T17:14:03.087 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-02-14T17:14:03.087 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-02-14T17:14:03.087 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-02-14T17:14:03.087 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-02-14T17:14:03.087 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-14T17:14:03.276 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: Writing manifest to image destination 2024-02-14T17:14:03.437 INFO:teuthology.orchestra.run.smithi017.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-02-14T17:14:03.878 INFO:teuthology.orchestra.run.smithi017.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-14T17:14:04.386 INFO:teuthology.orchestra.run.smithi017.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-02-14T17:14:04.386 INFO:teuthology.orchestra.run.smithi017.stderr:Extracting ceph user uid/gid from container image... 2024-02-14T17:14:04.769 INFO:teuthology.orchestra.run.smithi017.stderr:stat: 167 167 2024-02-14T17:14:05.324 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial keys... 2024-02-14T17:14:05.674 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQDd9MxlKR2RIxAAzJrvlxjzsqFgtUWi7dwHeQ== 2024-02-14T17:14:06.554 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQDe9MxlABV5HhAAiBaCRQgOvsopElYLjqC1Jg== 2024-02-14T17:14:07.318 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph-authtool: AQDf9MxlMjXlDRAA9hMPpkz1PH4+oIvcKlvdBQ== 2024-02-14T17:14:07.909 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial monmap... 2024-02-14T17:14:08.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-14T17:14:08.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-02-14T17:14:08.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:08.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-14T17:14:08.721 INFO:teuthology.orchestra.run.smithi017.stderr:monmaptool for a [v2:172.21.15.17:3300,v1:172.21.15.17:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-14T17:14:08.721 INFO:teuthology.orchestra.run.smithi017.stderr:setting min_mon_release = octopus 2024-02-14T17:14:08.721 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: set fsid to 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:08.721 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-14T17:14:08.722 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:14:08.722 INFO:teuthology.orchestra.run.smithi017.stderr:Creating mon... 2024-02-14T17:14:09.536 INFO:teuthology.orchestra.run.smithi017.stderr:create mon.a on 2024-02-14T17:14:09.738 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-14T17:14:09.874 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-612435b4-cb5c-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-612435b4-cb5c-11ee-95b9-87774f69a715.target. 2024-02-14T17:14:09.874 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-612435b4-cb5c-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-612435b4-cb5c-11ee-95b9-87774f69a715.target. 2024-02-14T17:14:10.162 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to reset failed state of unit ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service: Unit ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service not loaded. 2024-02-14T17:14:10.169 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph-612435b4-cb5c-11ee-95b9-87774f69a715.target.wants/ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-612435b4-cb5c-11ee-95b9-87774f69a715@.service. 2024-02-14T17:14:10.784 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:10 smithi017 podman[24392]: 2024-02-14 17:14:10.722883714 +0000 UTC m=+0.300884818 container start 189e2f6c0f7cdabd138dc850ab7cc4428753bc39ea3b7b4bf60c406947ae1082 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_CLEAN=True, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-14T17:14:10.785 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:10 smithi017 ceph-mon[24411]: mkfs 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:10.824 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-02-14T17:14:10.830 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-02-14T17:14:10.830 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-02-14T17:14:10.831 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to enable service . firewalld.service is not available 2024-02-14T17:14:10.831 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mon to start... 2024-02-14T17:14:10.831 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mon... 2024-02-14T17:14:11.038 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:10 smithi017 ceph-mon[24411]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:14:11.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:10 smithi017 bash[24392]: 189e2f6c0f7cdabd138dc850ab7cc4428753bc39ea3b7b4bf60c406947ae1082 2024-02-14T17:14:11.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:10 smithi017 systemd[1]: Started Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: cluster: 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: id: 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: services: 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.542174s) 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-14T17:14:11.326 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-14T17:14:11.327 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:11.327 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: data: 2024-02-14T17:14:11.327 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-14T17:14:11.327 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-14T17:14:11.327 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-14T17:14:11.327 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: pgs: 2024-02-14T17:14:11.327 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:mon is available 2024-02-14T17:14:11.771 INFO:teuthology.orchestra.run.smithi017.stderr:Assimilating anything we can from ceph.conf... 2024-02-14T17:14:12.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:11 smithi017 ceph-mon[24411]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:14:12.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:11 smithi017 ceph-mon[24411]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-02-14T17:14:12.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:11 smithi017 ceph-mon[24411]: fsmap 2024-02-14T17:14:12.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:11 smithi017 ceph-mon[24411]: osdmap e1: 0 total, 0 up, 0 in 2024-02-14T17:14:12.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:11 smithi017 ceph-mon[24411]: mgrmap e1: no daemons active 2024-02-14T17:14:12.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:11 smithi017 ceph-mon[24411]: from='client.? 172.21.15.17:0/3184173728' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:14:14.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:13 smithi017 ceph-mon[24411]: from='client.? 172.21.15.17:0/2021585474' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-14T17:14:14.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:13 smithi017 ceph-mon[24411]: from='client.? 172.21.15.17:0/2021585474' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [global] 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: fsid = 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.17:3300,v1:172.21.15.17:6789] 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [mgr] 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-14T17:14:14.541 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:14.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [osd] 2024-02-14T17:14:14.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-14T17:14:14.542 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-14T17:14:20.196 INFO:teuthology.orchestra.run.smithi017.stderr:Generating new minimal ceph.conf... 2024-02-14T17:14:23.595 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:23 smithi017 ceph-mon[24411]: from='client.? 172.21.15.17:0/3992302611' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:14:23.818 INFO:teuthology.orchestra.run.smithi017.stderr:Restarting the monitor... 2024-02-14T17:14:23.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:23 smithi017 systemd[1]: Stopping Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:14:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:23 smithi017 bash[24640]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-14T17:14:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a[24407]: 2024-02-14T17:14:23.975+0000 7f18425f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:14:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a[24407]: 2024-02-14T17:14:23.975+0000 7f18425f2700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-14T17:14:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:23 smithi017 podman[24653]: 2024-02-14 17:14:23.986981285 +0000 UTC m=+0.118283781 container died 189e2f6c0f7cdabd138dc850ab7cc4428753bc39ea3b7b4bf60c406947ae1082 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.buildah.version=1.19.8, name=centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, architecture=x86_64, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 podman[24653]: 2024-02-14 17:14:24.309195363 +0000 UTC m=+0.440497862 container cleanup 189e2f6c0f7cdabd138dc850ab7cc4428753bc39ea3b7b4bf60c406947ae1082 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, RELEASE=HEAD, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, release=754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 bash[24653]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 podman[24667]: 2024-02-14 17:14:24.451909348 +0000 UTC m=+0.460974081 container remove 189e2f6c0f7cdabd138dc850ab7cc4428753bc39ea3b7b4bf60c406947ae1082 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-type=git, io.buildah.version=1.19.8, RELEASE=HEAD, io.openshift.expose-services=, version=8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream) 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 bash[24681]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon.a" found: no such container 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 systemd[1]: Stopped Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 systemd[1]: Starting Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:14:24.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 podman[24750]: 2024-02-14 17:14:24.686158008 +0000 UTC m=+0.106403824 container create c7c59c3a31076f72a3622b516d5787e5f87d59813720a0039257beaf0d1ba3b3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, com.redhat.component=centos-stream-container, release=754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 podman[24750]: 2024-02-14 17:14:24.598126797 +0000 UTC m=+0.018372616 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 podman[24750]: 2024-02-14 17:14:24.896765958 +0000 UTC m=+0.317011775 container init c7c59c3a31076f72a3622b516d5787e5f87d59813720a0039257beaf0d1ba3b3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.openshift.expose-services=, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True) 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 podman[24750]: 2024-02-14 17:14:24.899511553 +0000 UTC m=+0.319757370 container start c7c59c3a31076f72a3622b516d5787e5f87d59813720a0039257beaf0d1ba3b3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, name=centos-stream, version=8, io.openshift.expose-services=, release=754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: set uid:gid to 167:167 (ceph:ceph) 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: pidfile_write: ignore empty --pid-file 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: load: jerasure load: lrc 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: RocksDB version: 6.15.5 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Compile date Apr 18 2022 2024-02-14T17:14:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: DB SUMMARY 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: DB Session ID: DCZF5C0OEYN50RB9NT9F 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83637 ; 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.env: 0x565319ec6860 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.fs: Posix File System 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.info_log: 0x56531b275dc0 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:14:24.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.statistics: (nil) 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.use_fsync: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.db_log_dir: 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-02-14T17:14:24.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:14:24.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:14:24.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:14:24.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:14:24.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.write_buffer_manager: 0x56531b3662a0 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:14:24.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.unordered_write: 0 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.row_cache: None 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.wal_filter: None 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:14:24.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.preserve_deletes: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:14:24.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_open_files: -1 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:14:24.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Compression algorithms supported: 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kZSTD supported: 0 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.merge_operator: 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_filter: None 2024-02-14T17:14:24.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56531b241d00) 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: hash_index_allow_collision: 1 2024-02-14T17:14:24.949 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 1 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x56531b2ac170 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2024-02-14T17:14:24.950 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 4 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression: NoCompression 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:14:24.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.num_levels: 7 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:14:24.952 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:14:24.953 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.arena_block_size: 4194304 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:14:24.954 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:14:24.955 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.ttl: 2592000 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707930864925007, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-14T17:14:24.956 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707930864925727, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79735, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77981, "index_size": 242, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10059, "raw_average_key_size": 47, "raw_value_size": 72245, "raw_average_value_size": 339, "num_data_blocks": 11, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707930864, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4c36e236-ef52-4870-b795-549bfe26281a", "db_session_id": "DCZF5C0OEYN50RB9NT9F"}} 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707930864926051, "job": 1, "event": "recovery_finished"} 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x56531b292a80 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: DB pointer 0x56531b2a2000 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:14:24.957 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 79.63 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 79.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:14:24.958 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 25.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 25.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:14:24.959 INFO:journalctl@ceph.mon.a.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 2/0 79.63 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 79.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:14:24.960 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 25.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???) e1 preinit fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???).mds e1 new map 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???).mds e1 print_map 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:14:24.961 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2024-02-14T17:14:24.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-14T17:14:24.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:14:24.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:14:24.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:14:24.962 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-14T17:14:24.994 INFO:teuthology.orchestra.run.smithi017.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-14T17:14:25.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 bash[24750]: c7c59c3a31076f72a3622b516d5787e5f87d59813720a0039257beaf0d1ba3b3 2024-02-14T17:14:25.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 systemd[1]: Started Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:14:25.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:14:25.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-02-14T17:14:25.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: fsmap 2024-02-14T17:14:25.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: osdmap e1: 0 total, 0 up, 0 in 2024-02-14T17:14:25.317 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:24 smithi017 ceph-mon[24769]: mgrmap e1: no daemons active 2024-02-14T17:14:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-14T17:14:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:14:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:Creating mgr... 2024-02-14T17:14:26.032 INFO:teuthology.orchestra.run.smithi017.stderr:Verifying port 9283 ... 2024-02-14T17:14:26.206 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Failed to reset failed state of unit ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service: Unit ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service not loaded. 2024-02-14T17:14:26.212 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: Created symlink /etc/systemd/system/ceph-612435b4-cb5c-11ee-95b9-87774f69a715.target.wants/ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-612435b4-cb5c-11ee-95b9-87774f69a715@.service. 2024-02-14T17:14:26.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:26 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/355402347' entity='client.admin' 2024-02-14T17:14:26.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:26 smithi017 podman[24987]: 2024-02-14 17:14:26.579176197 +0000 UTC m=+0.116201702 container create da74f8a06125b51f3b7df6cefce247369c7451c88923d6784f47e8db06c63437 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, release=754, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_BRANCH=HEAD, vcs-type=git, ceph=True, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-14T17:14:26.903 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-02-14T17:14:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-02-14T17:14:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-02-14T17:14:26.911 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to enable service . firewalld.service is not available 2024-02-14T17:14:26.921 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-02-14T17:14:26.929 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-02-14T17:14:26.930 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-02-14T17:14:26.930 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-14T17:14:26.930 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr to start... 2024-02-14T17:14:26.930 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr... 2024-02-14T17:14:27.055 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:26 smithi017 podman[24987]: 2024-02-14 17:14:26.789883327 +0000 UTC m=+0.326908835 container init da74f8a06125b51f3b7df6cefce247369c7451c88923d6784f47e8db06c63437 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , release=754, io.openshift.expose-services=, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vcs-type=git, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.buildah.version=1.19.8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-14T17:14:27.055 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:26 smithi017 podman[24987]: 2024-02-14 17:14:26.792362499 +0000 UTC m=+0.329388005 container start da74f8a06125b51f3b7df6cefce247369c7451c88923d6784f47e8db06c63437 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, distribution-scope=public, release=754, vcs-type=git, RELEASE=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0) 2024-02-14T17:14:27.055 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:26 smithi017 bash[24987]: da74f8a06125b51f3b7df6cefce247369c7451c88923d6784f47e8db06c63437 2024-02-14T17:14:27.056 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:26 smithi017 systemd[1]: Started Ceph mgr.y for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:14:27.519 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "612435b4-cb5c-11ee-95b9-87774f69a715", 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:27.520 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-14T17:14:27.521 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-14T17:14:27.525 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": false, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-02-14T17:14:27.526 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-02-14T17:14:10.782174+0000", 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-14T17:14:27.527 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-02-14T17:14:27.563 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:27 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:27.304+0000 7f728ae4d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:14:27.564 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:27 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:27.397+0000 7f728ae4d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:14:27.865 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:27 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/488785766' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:14:28.133 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.016+0000 7f728ae4d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:14:28.133 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.106+0000 7f728ae4d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:14:28.135 INFO:teuthology.orchestra.run.smithi017.stderr:mgr not available, waiting (1/15)... 2024-02-14T17:14:28.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.191+0000 7f728ae4d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:14:28.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.341+0000 7f728ae4d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:14:28.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.423+0000 7f728ae4d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:14:28.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.535+0000 7f728ae4d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:14:28.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.619+0000 7f728ae4d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:14:29.189 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.784+0000 7f728ae4d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:14:29.189 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:28.871+0000 7f728ae4d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:14:30.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:29 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:29.920+0000 7f728ae4d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:14:30.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:30.052+0000 7f728ae4d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:14:30.513 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:30.253+0000 7f728ae4d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:14:30.514 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:30.345+0000 7f728ae4d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:14:30.514 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:30.491+0000 7f728ae4d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:14:30.704 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:30.704 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "612435b4-cb5c-11ee-95b9-87774f69a715", 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:30.705 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-14T17:14:30.706 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-14T17:14:30.710 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": false, 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-02-14T17:14:30.711 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-02-14T17:14:10.782174+0000", 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-14T17:14:30.712 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-02-14T17:14:31.128 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/944828914' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:14:31.128 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:30.825+0000 7f728ae4d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:14:31.128 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:31.058+0000 7f728ae4d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:14:31.338 INFO:teuthology.orchestra.run.smithi017.stderr:mgr not available, waiting (2/15)... 2024-02-14T17:14:31.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:31.146+0000 7f728ae4d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:14:31.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:31.331+0000 7f728ae4d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:14:32.568 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:32.266+0000 7f728ae4d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:14:32.568 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:32.567+0000 7f728ae4d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:14:32.880 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:32.663+0000 7f728ae4d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:14:32.880 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:32.815+0000 7f728ae4d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:14:33.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: Activating manager daemon y 2024-02-14T17:14:33.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: mgrmap e2: y(active, starting, since 0.00285165s) 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: Manager daemon y is now available 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' 2024-02-14T17:14:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:32 smithi017 ceph-mon[24769]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' 2024-02-14T17:14:33.842 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:33.842 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-02-14T17:14:33.842 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsid": "612435b4-cb5c-11ee-95b9-87774f69a715", 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "health": { 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "checks": {}, 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mutes": [] 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum": [ 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 0 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "a" 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-14T17:14:33.843 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "monmap": { 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osdmap": { 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-14T17:14:33.844 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgmap": { 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "fsmap": { 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-14T17:14:33.845 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modules": [ 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "iostat", 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "nfs", 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "restful" 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ], 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "servicemap": { 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "modified": "2024-02-14T17:14:10.782174+0000", 2024-02-14T17:14:33.846 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "services": {} 2024-02-14T17:14:33.847 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: }, 2024-02-14T17:14:33.847 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-14T17:14:33.847 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-02-14T17:14:34.636 INFO:teuthology.orchestra.run.smithi017.stderr:mgr is available 2024-02-14T17:14:35.012 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:34 smithi017 ceph-mon[24769]: mgrmap e3: y(active, since 1.00585s) 2024-02-14T17:14:35.012 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:34 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/264737014' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [global] 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: fsid = 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [mgr] 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: [osd] 2024-02-14T17:14:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-14T17:14:35.250 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-14T17:14:35.708 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling cephadm module... 2024-02-14T17:14:36.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:35 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1778392115' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-14T17:14:36.100 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:35 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1778392115' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-14T17:14:36.101 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:35 smithi017 ceph-mon[24769]: mgrmap e4: y(active, since 3s) 2024-02-14T17:14:37.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:36 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1491643096' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-14T17:14:37.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setuser ceph since I am not root 2024-02-14T17:14:37.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setgroup ceph since I am not root 2024-02-14T17:14:37.503 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:37 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:37.445+0000 7f79c92ba000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:14:37.799 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:37 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:37.539+0000 7f79c92ba000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:14:38.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:37 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1491643096' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-14T17:14:38.136 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:37 smithi017 ceph-mon[24769]: mgrmap e5: y(active, since 4s) 2024-02-14T17:14:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-02-14T17:14:38.137 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-14T17:14:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-02-14T17:14:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-14T17:14:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-14T17:14:38.138 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-02-14T17:14:38.439 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.154+0000 7f79c92ba000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:14:38.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.246+0000 7f79c92ba000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:14:38.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.330+0000 7f79c92ba000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:14:38.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.483+0000 7f79c92ba000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:14:38.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.565+0000 7f79c92ba000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:14:38.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.676+0000 7f79c92ba000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:14:38.762 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.761+0000 7f79c92ba000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:14:38.771 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for the mgr to restart... 2024-02-14T17:14:38.771 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr epoch 5... 2024-02-14T17:14:39.018 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1579533481' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-14T17:14:39.019 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:38.929+0000 7f79c92ba000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:14:39.019 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:39 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:39.018+0000 7f79c92ba000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:14:40.388 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:40.062+0000 7f79c92ba000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:14:40.388 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:40.193+0000 7f79c92ba000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:14:40.388 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:40.387+0000 7f79c92ba000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:14:40.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:40.475+0000 7f79c92ba000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:14:40.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:40.614+0000 7f79c92ba000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:14:41.162 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:40.936+0000 7f79c92ba000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:14:41.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:41.161+0000 7f79c92ba000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:14:41.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:41.246+0000 7f79c92ba000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:14:41.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:41.423+0000 7f79c92ba000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:14:42.659 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:42.356+0000 7f79c92ba000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:14:42.659 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:42.657+0000 7f79c92ba000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:14:42.939 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:42.753+0000 7f79c92ba000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:14:42.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:42.906+0000 7f79c92ba000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:14:43.982 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-02-14T17:14:43.983 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-14T17:14:43.983 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "initialized": true 2024-02-14T17:14:43.983 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-02-14T17:14:44.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: Active manager daemon y restarted 2024-02-14T17:14:44.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: Activating manager daemon y 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: osdmap e2: 0 total, 0 up, 0 in 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: mgrmap e6: y(active, starting, since 0.0528717s) 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: Manager daemon y is now available 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:14:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:14:44.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:43 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:14:44.510 INFO:teuthology.orchestra.run.smithi017.stderr:mgr epoch 5 is available 2024-02-14T17:14:44.510 INFO:teuthology.orchestra.run.smithi017.stderr:Setting orchestrator backend to cephadm... 2024-02-14T17:14:45.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:44 smithi017 ceph-mon[24769]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-14T17:14:45.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:44 smithi017 ceph-mon[24769]: mgrmap e7: y(active, since 1.05513s) 2024-02-14T17:14:45.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:44 smithi017 ceph-mon[24769]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-14T17:14:45.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:14:45] ENGINE Bus STARTING 2024-02-14T17:14:45.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:14:45] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:14:45.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:14:45] ENGINE Bus STARTED 2024-02-14T17:14:46.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-mon[24769]: mgrmap e8: y(active, since 2s) 2024-02-14T17:14:46.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:46.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:14:46.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:46.050 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:45 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:14:46.338 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: value unchanged 2024-02-14T17:14:46.890 INFO:teuthology.orchestra.run.smithi017.stderr:Generating ssh key... 2024-02-14T17:14:47.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:46 smithi017 ceph-mon[24769]: [14/Feb/2024:17:14:45] ENGINE Bus STARTING 2024-02-14T17:14:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:46 smithi017 ceph-mon[24769]: [14/Feb/2024:17:14:45] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:14:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:46 smithi017 ceph-mon[24769]: [14/Feb/2024:17:14:45] ENGINE Bus STARTED 2024-02-14T17:14:48.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:47 smithi017 ceph-mon[24769]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: Generating public/private rsa key pair. 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: Your identification has been saved in /tmp/tmp74sz8cqb/key. 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: Your public key has been saved in /tmp/tmp74sz8cqb/key.pub. 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: The key fingerprint is: 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: SHA256:oob6VRI/LQiy6hvBW1P2P/UxZFdhSd4flWAfGo7BhVI ceph-612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: The key's randomart image is: 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: +---[RSA 3072]----+ 2024-02-14T17:14:48.443 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: | oEo=o=*| 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: | . o= *o=| 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: |. . .o .. = =.| 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: |.o .o+.. o . o| 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: |.o oo *.S . o .| 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: |. +..+ +. . . o | 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: |.o. + o . | 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: |...o . | 2024-02-14T17:14:48.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: |.+o | 2024-02-14T17:14:48.445 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: +----[SHA256]-----+ 2024-02-14T17:14:49.306 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-mon[24769]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:14:49.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-mon[24769]: Generating ssh key... 2024-02-14T17:14:49.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:49.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:48 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:49.307 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8dZj6tyDChGnj+QezQiKn4OEbCXnUhVHqFVfb8vVLkdWb4sBQgAVYFJmI2uQZgzMZfDfJunTcM2dogCWTr+0hrZkM/upfcYgzyYWsu1NomjKD/LHFCaqhC088skgU5hJGOn4n3uksBtjtFGwq/mgqSZalwE3NFVXFcBFeSQDWImupShK2CtI6pxUu82kzLA6byPRTlc8fYmmSDnKi0/auKnZzJ+jsoBI8F4PCb9vtJhoruIUzWeYLVkQTtAPNaKLdGWB/VQPtf9wsSD8/EYNMJl8xuKtEzSIdooioL1wo5ZgD5IxYSHvjPWoy2XuP5Dt0MZOKjjnOZiRBQW/XpAq1ahC73UJ6nlPsXdHwz+i9WMjUeWZiATDr8YnP+NSH2GZC4vVdmE/mJQPwABxHOfsgo0B66xsflaHUf3YX9Pe/z2edJcPciK1OnFw/YXPD+dnjut9lXK5NbUOME/inxEuasOhvTbnbzeKFewsfR/gPdYfTQIdkKqj5sLitwyqBhZE= ceph-612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:14:49.827 INFO:teuthology.orchestra.run.smithi017.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-14T17:14:49.827 INFO:teuthology.orchestra.run.smithi017.stderr:Adding key to root@localhost authorized_keys... 2024-02-14T17:14:49.827 INFO:teuthology.orchestra.run.smithi017.stderr:Adding host smithi017... 2024-02-14T17:14:51.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:50 smithi017 ceph-mon[24769]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:14:51.504 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Added host 'smithi017' with addr '172.21.15.17' 2024-02-14T17:14:52.184 INFO:teuthology.orchestra.run.smithi017.stderr:Deploying unmanaged mon service... 2024-02-14T17:14:52.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:51 smithi017 ceph-mon[24769]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi017", "addr": "172.21.15.17", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:14:52.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:51 smithi017 ceph-mon[24769]: Deploying cephadm binary to smithi017 2024-02-14T17:14:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:51 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:51 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:14:52.814 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-14T17:14:53.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:52 smithi017 ceph-mon[24769]: Added host smithi017 2024-02-14T17:14:53.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:52 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:53.367 INFO:teuthology.orchestra.run.smithi017.stderr:Deploying unmanaged mgr service... 2024-02-14T17:14:54.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:53 smithi017 ceph-mon[24769]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:14:54.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:53 smithi017 ceph-mon[24769]: Saving service mon spec with placement count:5 2024-02-14T17:14:54.296 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-14T17:14:55.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:55 smithi017 ceph-mon[24769]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:14:55.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:55 smithi017 ceph-mon[24769]: Saving service mgr spec with placement count:2 2024-02-14T17:14:55.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:55 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:56.819 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:56 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3049612926' entity='client.admin' 2024-02-14T17:14:56.819 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:56 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:56.819 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:56 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:58.155 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling the dashboard module... 2024-02-14T17:14:58.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:58 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3605044715' entity='client.admin' 2024-02-14T17:14:58.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:58 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:59.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:59 smithi017 ceph-mon[24769]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:14:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:14:59 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3756173644' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-14T17:14:59.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setuser ceph since I am not root 2024-02-14T17:14:59.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setgroup ceph since I am not root 2024-02-14T17:15:00.099 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:59.829+0000 7f3b21077000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:15:00.099 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:14:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:14:59.923+0000 7f3b21077000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:15:00.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:00 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3756173644' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-14T17:15:00.434 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:00 smithi017 ceph-mon[24769]: mgrmap e9: y(active, since 16s) 2024-02-14T17:15:00.435 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-02-14T17:15:00.435 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "epoch": 9, 2024-02-14T17:15:00.435 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "available": true, 2024-02-14T17:15:00.435 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "active_name": "y", 2024-02-14T17:15:00.435 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-14T17:15:00.435 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-02-14T17:15:00.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:00.547+0000 7f3b21077000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:15:00.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:00.639+0000 7f3b21077000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:15:01.072 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:00.724+0000 7f3b21077000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:15:01.072 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:00.876+0000 7f3b21077000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:15:01.072 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:00.959+0000 7f3b21077000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:15:01.072 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:01.070+0000 7f3b21077000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:15:01.166 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for the mgr to restart... 2024-02-14T17:15:01.166 INFO:teuthology.orchestra.run.smithi017.stderr:Waiting for mgr epoch 9... 2024-02-14T17:15:01.324 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:01.155+0000 7f3b21077000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:15:01.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:01 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3132193167' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-14T17:15:01.616 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:01.326+0000 7f3b21077000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:15:01.616 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:01.413+0000 7f3b21077000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:15:02.807 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:02.477+0000 7f3b21077000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:15:02.807 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:02.609+0000 7f3b21077000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:15:02.807 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:02.805+0000 7f3b21077000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:15:03.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:02.893+0000 7f3b21077000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:15:03.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:03.031+0000 7f3b21077000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:15:03.665 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:03.352+0000 7f3b21077000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:15:03.665 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:03.577+0000 7f3b21077000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:15:03.665 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:03.663+0000 7f3b21077000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:15:03.939 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:03.836+0000 7f3b21077000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:15:05.069 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:04.767+0000 7f3b21077000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:15:05.069 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:05 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:05.067+0000 7f3b21077000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:15:05.439 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:05 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:05.164+0000 7f3b21077000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:15:05.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:05 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:15:05.316+0000 7f3b21077000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:15:06] ENGINE Bus STARTING 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:15:06] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:15:06] ENGINE Bus STARTED 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: Active manager daemon y restarted 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: Activating manager daemon y 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: osdmap e3: 0 total, 0 up, 0 in 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: mgrmap e10: y(active, starting, since 0.608866s) 2024-02-14T17:15:06.679 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: Manager daemon y is now available 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:15:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:15:06.949 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: { 2024-02-14T17:15:06.950 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-02-14T17:15:06.950 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: "initialized": true 2024-02-14T17:15:06.950 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: } 2024-02-14T17:15:07.501 INFO:teuthology.orchestra.run.smithi017.stderr:mgr epoch 9 is available 2024-02-14T17:15:07.501 INFO:teuthology.orchestra.run.smithi017.stderr:Generating a dashboard self-signed certificate... 2024-02-14T17:15:07.851 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:07 smithi017 ceph-mon[24769]: [14/Feb/2024:17:15:06] ENGINE Bus STARTING 2024-02-14T17:15:07.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:07 smithi017 ceph-mon[24769]: [14/Feb/2024:17:15:06] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:15:07.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:07 smithi017 ceph-mon[24769]: [14/Feb/2024:17:15:06] ENGINE Bus STARTED 2024-02-14T17:15:07.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:07 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:07.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:07 smithi017 ceph-mon[24769]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-14T17:15:07.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:07 smithi017 ceph-mon[24769]: mgrmap e11: y(active, since 1.61137s) 2024-02-14T17:15:07.852 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:07 smithi017 ceph-mon[24769]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-14T17:15:08.213 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-14T17:15:08.774 INFO:teuthology.orchestra.run.smithi017.stderr:Creating initial admin user... 2024-02-14T17:15:09.024 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:08 smithi017 ceph-mon[24769]: mgrmap e12: y(active, since 2s) 2024-02-14T17:15:09.025 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:08 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:09.025 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:08 smithi017 ceph-mon[24769]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:09.025 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:08 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:09.025 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:08 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:09.025 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:08 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:09.691 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$XevjrIv8NJdXFqQG2gD6fOFTwiuZkmaC.umW8O7JdvW00ejYWjugK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707930909, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-14T17:15:10.394 INFO:teuthology.orchestra.run.smithi017.stderr:Fetching dashboard port number... 2024-02-14T17:15:10.918 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:10 smithi017 ceph-mon[24769]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:10.918 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:10 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:11.463 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: 8443 2024-02-14T17:15:11.784 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:11 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2684329463' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-14T17:15:12.096 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: disabled 2024-02-14T17:15:12.103 INFO:teuthology.orchestra.run.smithi017.stderr:systemctl: inactive 2024-02-14T17:15:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:firewalld.service is not enabled 2024-02-14T17:15:12.104 INFO:teuthology.orchestra.run.smithi017.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-14T17:15:12.106 INFO:teuthology.orchestra.run.smithi017.stderr:Ceph Dashboard is now available at: 2024-02-14T17:15:12.107 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:12.107 INFO:teuthology.orchestra.run.smithi017.stderr: URL: https://smithi017.front.sepia.ceph.com:8443/ 2024-02-14T17:15:12.107 INFO:teuthology.orchestra.run.smithi017.stderr: User: admin 2024-02-14T17:15:12.107 INFO:teuthology.orchestra.run.smithi017.stderr: Password: ddl6ei8cvm 2024-02-14T17:15:12.107 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:12.107 INFO:teuthology.orchestra.run.smithi017.stderr:Enabling autotune for osd_memory_target 2024-02-14T17:15:13.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:12 smithi017 ceph-mon[24769]: mgrmap e13: y(active, since 6s) 2024-02-14T17:15:14.067 INFO:teuthology.orchestra.run.smithi017.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-02-14T17:15:14.702 INFO:teuthology.orchestra.run.smithi017.stderr:You can access the Ceph CLI with: 2024-02-14T17:15:14.702 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:14.702 INFO:teuthology.orchestra.run.smithi017.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:14.702 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr: ceph telemetry on 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr:For more information see: 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:15:14.703 INFO:teuthology.orchestra.run.smithi017.stderr:Bootstrap complete. 2024-02-14T17:15:14.741 INFO:tasks.cephadm:Fetching config... 2024-02-14T17:15:14.742 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:15:14.742 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-14T17:15:14.759 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-14T17:15:14.759 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:15:14.759 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-14T17:15:14.814 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-14T17:15:14.814 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:15:14.814 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-14T17:15:14.882 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-14T17:15:14.882 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:15:14.882 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-14T17:15:14.937 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-14T17:15:14.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8dZj6tyDChGnj+QezQiKn4OEbCXnUhVHqFVfb8vVLkdWb4sBQgAVYFJmI2uQZgzMZfDfJunTcM2dogCWTr+0hrZkM/upfcYgzyYWsu1NomjKD/LHFCaqhC088skgU5hJGOn4n3uksBtjtFGwq/mgqSZalwE3NFVXFcBFeSQDWImupShK2CtI6pxUu82kzLA6byPRTlc8fYmmSDnKi0/auKnZzJ+jsoBI8F4PCb9vtJhoruIUzWeYLVkQTtAPNaKLdGWB/VQPtf9wsSD8/EYNMJl8xuKtEzSIdooioL1wo5ZgD5IxYSHvjPWoy2XuP5Dt0MZOKjjnOZiRBQW/XpAq1ahC73UJ6nlPsXdHwz+i9WMjUeWZiATDr8YnP+NSH2GZC4vVdmE/mJQPwABxHOfsgo0B66xsflaHUf3YX9Pe/z2edJcPciK1OnFw/YXPD+dnjut9lXK5NbUOME/inxEuasOhvTbnbzeKFewsfR/gPdYfTQIdkKqj5sLitwyqBhZE= ceph-612435b4-cb5c-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-14T17:15:15.036 INFO:teuthology.orchestra.run.smithi017.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8dZj6tyDChGnj+QezQiKn4OEbCXnUhVHqFVfb8vVLkdWb4sBQgAVYFJmI2uQZgzMZfDfJunTcM2dogCWTr+0hrZkM/upfcYgzyYWsu1NomjKD/LHFCaqhC088skgU5hJGOn4n3uksBtjtFGwq/mgqSZalwE3NFVXFcBFeSQDWImupShK2CtI6pxUu82kzLA6byPRTlc8fYmmSDnKi0/auKnZzJ+jsoBI8F4PCb9vtJhoruIUzWeYLVkQTtAPNaKLdGWB/VQPtf9wsSD8/EYNMJl8xuKtEzSIdooioL1wo5ZgD5IxYSHvjPWoy2XuP5Dt0MZOKjjnOZiRBQW/XpAq1ahC73UJ6nlPsXdHwz+i9WMjUeWZiATDr8YnP+NSH2GZC4vVdmE/mJQPwABxHOfsgo0B66xsflaHUf3YX9Pe/z2edJcPciK1OnFw/YXPD+dnjut9lXK5NbUOME/inxEuasOhvTbnbzeKFewsfR/gPdYfTQIdkKqj5sLitwyqBhZE= ceph-612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:15:15.052 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8dZj6tyDChGnj+QezQiKn4OEbCXnUhVHqFVfb8vVLkdWb4sBQgAVYFJmI2uQZgzMZfDfJunTcM2dogCWTr+0hrZkM/upfcYgzyYWsu1NomjKD/LHFCaqhC088skgU5hJGOn4n3uksBtjtFGwq/mgqSZalwE3NFVXFcBFeSQDWImupShK2CtI6pxUu82kzLA6byPRTlc8fYmmSDnKi0/auKnZzJ+jsoBI8F4PCb9vtJhoruIUzWeYLVkQTtAPNaKLdGWB/VQPtf9wsSD8/EYNMJl8xuKtEzSIdooioL1wo5ZgD5IxYSHvjPWoy2XuP5Dt0MZOKjjnOZiRBQW/XpAq1ahC73UJ6nlPsXdHwz+i9WMjUeWZiATDr8YnP+NSH2GZC4vVdmE/mJQPwABxHOfsgo0B66xsflaHUf3YX9Pe/z2edJcPciK1OnFw/YXPD+dnjut9lXK5NbUOME/inxEuasOhvTbnbzeKFewsfR/gPdYfTQIdkKqj5sLitwyqBhZE= ceph-612435b4-cb5c-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-14T17:15:15.098 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8dZj6tyDChGnj+QezQiKn4OEbCXnUhVHqFVfb8vVLkdWb4sBQgAVYFJmI2uQZgzMZfDfJunTcM2dogCWTr+0hrZkM/upfcYgzyYWsu1NomjKD/LHFCaqhC088skgU5hJGOn4n3uksBtjtFGwq/mgqSZalwE3NFVXFcBFeSQDWImupShK2CtI6pxUu82kzLA6byPRTlc8fYmmSDnKi0/auKnZzJ+jsoBI8F4PCb9vtJhoruIUzWeYLVkQTtAPNaKLdGWB/VQPtf9wsSD8/EYNMJl8xuKtEzSIdooioL1wo5ZgD5IxYSHvjPWoy2XuP5Dt0MZOKjjnOZiRBQW/XpAq1ahC73UJ6nlPsXdHwz+i9WMjUeWZiATDr8YnP+NSH2GZC4vVdmE/mJQPwABxHOfsgo0B66xsflaHUf3YX9Pe/z2edJcPciK1OnFw/YXPD+dnjut9lXK5NbUOME/inxEuasOhvTbnbzeKFewsfR/gPdYfTQIdkKqj5sLitwyqBhZE= ceph-612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:15:15.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-14T17:15:15.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:15 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2319728464' entity='client.admin' 2024-02-14T17:15:17.916 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-14T17:15:17.916 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-14T17:15:18.432 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:18 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1923418840' entity='client.admin' 2024-02-14T17:15:20.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:20.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:15:20.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:20.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:20.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:21.232 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi111 2024-02-14T17:15:21.232 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-14T17:15:21.232 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.conf 2024-02-14T17:15:21.250 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-14T17:15:21.251 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:21.308 INFO:tasks.cephadm:Adding host smithi111 to orchestrator... 2024-02-14T17:15:21.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch host add smithi111 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:21.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:21 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:23.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:23 smithi017 ceph-mon[24769]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:24.111 INFO:teuthology.orchestra.run.smithi017.stdout:Added host 'smithi111' with addr '172.21.15.111' 2024-02-14T17:15:24.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-14T17:15:25.277 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: Deploying cephadm binary to smithi111 2024-02-14T17:15:25.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:25.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: Added host smithi111 2024-02-14T17:15:25.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:25.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:25.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:25.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:25.278 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:26.681 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:15:26.682 INFO:teuthology.orchestra.run.smithi017.stdout:[{"addr": "172.21.15.17", "hostname": "smithi017", "labels": [], "status": ""}, {"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}] 2024-02-14T17:15:26.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:26 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:26.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:26 smithi017 ceph-mon[24769]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:27.249 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-14T17:15:27.250 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-14T17:15:27.802 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:27 smithi017 ceph-mon[24769]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:15:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:28 smithi017 ceph-mon[24769]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:15:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:29.738 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2024-02-14T17:15:30.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:29 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3850148441' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-14T17:15:30.376 INFO:tasks.cephadm:Adding mon.a on smithi017 2024-02-14T17:15:30.376 INFO:tasks.cephadm:Adding mon.c on smithi017 2024-02-14T17:15:30.376 INFO:tasks.cephadm:Adding mon.b on smithi111 2024-02-14T17:15:30.376 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch apply mon '3;smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi111:172.21.15.111=b' 2024-02-14T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3850148441' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-14T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:30 smithi017 ceph-mon[24769]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:31.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:30 smithi017 ceph-mon[24769]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:32.287 INFO:teuthology.orchestra.run.smithi111.stdout:Scheduled mon update... 2024-02-14T17:15:32.977 DEBUG:teuthology.orchestra.run.smithi017:mon.c> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.c.service 2024-02-14T17:15:32.979 DEBUG:teuthology.orchestra.run.smithi111:mon.b> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.b.service 2024-02-14T17:15:32.982 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-14T17:15:32.982 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi111:172.21.15.111=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi111:172.21.15.111=b;count:3 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:33.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:33.551 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:33.551 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:34.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:34 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:34.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:34 smithi017 ceph-mon[24769]: Deploying daemon mon.c on smithi017 2024-02-14T17:15:34.922 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:15:34.922 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":1,"fsid":"612435b4-cb5c-11ee-95b9-87774f69a715","modified":"2024-02-14T17:14:08.221365Z","created":"2024-02-14T17:14:08.221365Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-14T17:15:34.924 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 1 2024-02-14T17:15:35.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[24769]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:35.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/1387750136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:15:35.809 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 podman[29488]: 2024-02-14 17:15:35.701538674 +0000 UTC m=+0.116857110 container create 2ee2c64b201d24dce24f88c1ab5d4e01496885ce6bab980e2cdbfa745b1b1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, version=8, vcs-type=git, distribution-scope=public, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754) 2024-02-14T17:15:36.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 podman[29488]: 2024-02-14 17:15:35.928607812 +0000 UTC m=+0.343926206 container init 2ee2c64b201d24dce24f88c1ab5d4e01496885ce6bab980e2cdbfa745b1b1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, distribution-scope=public, io.buildah.version=1.19.8, GIT_CLEAN=True, ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container) 2024-02-14T17:15:36.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 podman[29488]: 2024-02-14 17:15:35.931563383 +0000 UTC m=+0.346881778 container start 2ee2c64b201d24dce24f88c1ab5d4e01496885ce6bab980e2cdbfa745b1b1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, release=754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-type=git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container) 2024-02-14T17:15:36.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: set uid:gid to 167:167 (ceph:ceph) 2024-02-14T17:15:36.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-02-14T17:15:36.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: pidfile_write: ignore empty --pid-file 2024-02-14T17:15:36.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: load: jerasure load: lrc 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: RocksDB version: 6.15.5 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Compile date Apr 18 2022 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: DB SUMMARY 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: DB Session ID: US2T4AKUYB4W4E5FASQY 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-14T17:15:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.env: 0x55a9238c6860 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.fs: Posix File System 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.info_log: 0x55a92503fdc0 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.statistics: (nil) 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.use_fsync: 0 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:15:36.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.db_log_dir: 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:15:36.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.write_buffer_manager: 0x55a9251302d0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.unordered_write: 0 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:15:36.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.row_cache: None 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.wal_filter: None 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.preserve_deletes: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:15:36.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:15:36.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:15:36.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:15:36.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:15:36.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:15:36.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:15:36.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:15:36.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_open_files: -1 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:15:36.198 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Compression algorithms supported: 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kZSTD supported: 0 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.merge_operator: 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_filter: None 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:15:36.199 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a92500bd00) 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_type: 0 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_index_type: 0 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_shortening: 1 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: hash_index_allow_collision: 1 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: checksum: 1 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: no_block_cache: 0 2024-02-14T17:15:36.200 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache: 0x55a925076170 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_options: 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: capacity : 536870912 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: num_shard_bits : 4 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: strict_capacity_limit : 0 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_compressed: (nil) 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: persistent_cache: (nil) 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size: 4096 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size_deviation: 10 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_restart_interval: 16 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_block_restart_interval: 1 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: metadata_block_size: 4096 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: partition_filters: 0 2024-02-14T17:15:36.201 INFO:journalctl@ceph.mon.c.smithi017.stdout: use_delta_encoding: 1 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout: whole_key_filtering: 1 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout: verify_compression: 0 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout: format_version: 4 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_index_compression: 1 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_align: 0 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression: NoCompression 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.num_levels: 7 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:15:36.202 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:15:36.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:15:36.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.arena_block_size: 4194304 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:15:36.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:15:36.206 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.ttl: 2592000 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707930935957342, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707930935957645, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707930935, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6ff770d3-8c5a-4162-a004-4f462f2c1f2b", "db_session_id": "US2T4AKUYB4W4E5FASQY"}} 2024-02-14T17:15:36.207 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707930935957961, "job": 1, "event": "recovery_finished"} 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55a92505ca80 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: DB pointer 0x55a92506c000 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** DB Stats ** 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:15:36.208 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:15:36.209 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:15:36.210 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-14T17:15:36.211 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: using public_addrv [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: starting mon.c rank -1 at public addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] at bind addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(???) e0 preinit fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).mds e1 new map 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout: e1 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout: legacy client fscid: -1 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout: No filesystems configured 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-14T17:15:36.212 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mkfs 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: fsmap 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: osdmap e1: 0 total, 0 up, 0 in 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e1: no daemons active 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3184173728' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:15:36.213 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2021585474' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2021585474' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3992302611' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: monmap e1: 1 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0]} 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: fsmap 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: osdmap e1: 0 total, 0 up, 0 in 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e1: no daemons active 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/355402347' entity='client.admin' 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/488785766' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/944828914' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Activating manager daemon y 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e2: y(active, starting, since 0.00285165s) 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:15:36.214 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Manager daemon y is now available 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14100 172.21.15.17:0/910498961' entity='mgr.y' 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e3: y(active, since 1.00585s) 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/264737014' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1778392115' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1778392115' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e4: y(active, since 3s) 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1491643096' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1491643096' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-14T17:15:36.215 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e5: y(active, since 4s) 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1579533481' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Active manager daemon y restarted 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Activating manager daemon y 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: osdmap e2: 0 total, 0 up, 0 in 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e6: y(active, starting, since 0.0528717s) 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Manager daemon y is now available 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e7: y(active, since 1.05513s) 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e8: y(active, since 2s) 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: [14/Feb/2024:17:14:45] ENGINE Bus STARTING 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: [14/Feb/2024:17:14:45] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: [14/Feb/2024:17:14:45] ENGINE Bus STARTED 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Generating ssh key... 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi017", "addr": "172.21.15.17", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Deploying cephadm binary to smithi017 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Added host smithi017 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Saving service mon spec with placement count:5 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Saving service mgr spec with placement count:2 2024-02-14T17:15:36.218 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3049612926' entity='client.admin' 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3605044715' entity='client.admin' 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14118 172.21.15.17:0/34209130' entity='mgr.y' 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3756173644' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3756173644' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e9: y(active, since 16s) 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3132193167' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Active manager daemon y restarted 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Activating manager daemon y 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: osdmap e3: 0 total, 0 up, 0 in 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e10: y(active, starting, since 0.608866s) 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:36.219 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Manager daemon y is now available 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: [14/Feb/2024:17:15:06] ENGINE Bus STARTING 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: [14/Feb/2024:17:15:06] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: [14/Feb/2024:17:15:06] ENGINE Bus STARTED 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e11: y(active, since 1.61137s) 2024-02-14T17:15:36.220 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e12: y(active, since 2s) 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2684329463' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mgrmap e13: y(active, since 6s) 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2319728464' entity='client.admin' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1923418840' entity='client.admin' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi017", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.221 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Deploying cephadm binary to smithi111 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Added host smithi111 2024-02-14T17:15:36.222 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3850148441' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3850148441' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:36.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi111:172.21.15.111=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Saving service mon spec with placement smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi111:172.21.15.111=b;count:3 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: Deploying daemon mon.c on smithi017 2024-02-14T17:15:36.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:36.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/1387750136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:15:36.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:35 smithi017 ceph-mon[29506]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-14T17:15:36.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:36 smithi017 bash[29488]: 2ee2c64b201d24dce24f88c1ab5d4e01496885ce6bab980e2cdbfa745b1b1b77 2024-02-14T17:15:36.225 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:36 smithi017 systemd[1]: Started Ceph mon.c for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:15:36.506 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-14T17:15:36.507 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: mon.a calling monitor election 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: mon.c calling monitor election 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: fsmap 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: mgrmap e13: y(active, since 35s) 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: overall HEALTH_OK 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:41.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: mon.a calling monitor election 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: mon.c calling monitor election 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: monmap e2: 2 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: fsmap 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: mgrmap e13: y(active, since 35s) 2024-02-14T17:15:41.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: overall HEALTH_OK 2024-02-14T17:15:41.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:41.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:41.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:41 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:41.475 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:15:41.475 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":2,"fsid":"612435b4-cb5c-11ee-95b9-87774f69a715","modified":"2024-02-14T17:15:35.976642Z","created":"2024-02-14T17:14:08.221365Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-14T17:15:41.476 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 2 2024-02-14T17:15:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:42 smithi017 ceph-mon[29506]: Deploying daemon mon.b on smithi111 2024-02-14T17:15:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:42 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/517619182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:15:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:42 smithi017 ceph-mon[24769]: Deploying daemon mon.b on smithi111 2024-02-14T17:15:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:42 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/517619182' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-14T17:15:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:42.992 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-14T17:15:42.993 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-14T17:15:43.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:43 smithi017 ceph-mon[29506]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:43.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:43 smithi017 ceph-mon[24769]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:45.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:45 smithi017 ceph-mon[24769]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:45.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:45 smithi017 ceph-mon[29506]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:45.800 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:15:45.800 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":2,"fsid":"612435b4-cb5c-11ee-95b9-87774f69a715","modified":"2024-02-14T17:15:35.976642Z","created":"2024-02-14T17:14:08.221365Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-14T17:15:45.801 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 2 2024-02-14T17:15:45.901 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:45 smithi111 ceph-mon[25603]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-02-14T17:15:46.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:45 smithi111 bash[25539]: 4257ce96e8cf94dda8fe5bc7eb7d7fb511525e72ceab6ecc88784502cb8a0134 2024-02-14T17:15:46.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:45 smithi111 systemd[1]: Started Ceph mon.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:15:47.551 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-14T17:15:47.551 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-14T17:15:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: mon.c calling monitor election 2024-02-14T17:15:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: mon.a calling monitor election 2024-02-14T17:15:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: fsmap 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: mgrmap e13: y(active, since 45s) 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:15:51.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: mon.b (rank 2) addr [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] is down (out of quorum) 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: mon.c calling monitor election 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: mon.a calling monitor election 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: fsmap 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: mgrmap e13: y(active, since 45s) 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:15:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: mon.b (rank 2) addr [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] is down (out of quorum) 2024-02-14T17:15:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:52.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:52.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:52.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:52.216 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:52.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:52.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:52.216 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:52.217 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:52.410 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-14T17:15:52.410 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":3,"fsid":"612435b4-cb5c-11ee-95b9-87774f69a715","modified":"2024-02-14T17:15:45.900425Z","created":"2024-02-14T17:14:08.221365Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3300","nonce":0},{"type":"v1","addr":"172.21.15.17:6789","nonce":0}]},"addr":"172.21.15.17:6789/0","public_addr":"172.21.15.17:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:3301","nonce":0},{"type":"v1","addr":"172.21.15.17:6790","nonce":0}]},"addr":"172.21.15.17:6790/0","public_addr":"172.21.15.17:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-14T17:15:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:dumped monmap epoch 3 2024-02-14T17:15:52.907 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-14T17:15:52.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph config generate-minimal-conf 2024-02-14T17:15:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: mon.c calling monitor election 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: mon.a calling monitor election 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: fsmap 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: mgrmap e13: y(active, since 45s) 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: mon.b (rank 2) addr [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] is down (out of quorum) 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:15:53.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:53.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:15:53.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:53.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:53.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:53.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:52 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: mon.b calling monitor election 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: mon.b calling monitor election 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: mon.c calling monitor election 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: mon.a calling monitor election 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: fsmap 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: mgrmap e13: y(active, since 47s) 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-14T17:15:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: Cluster is now healthy 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: overall HEALTH_OK 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: mon.b calling monitor election 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: mon.b calling monitor election 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: mon.c calling monitor election 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: mon.a calling monitor election 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: fsmap 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: mgrmap e13: y(active, since 47s) 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: Cluster is now healthy 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: overall HEALTH_OK 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:15:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:53 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: mon.b calling monitor election 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: mon.b calling monitor election 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: mon.c calling monitor election 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: mon.a calling monitor election 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: fsmap 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: osdmap e4: 0 total, 0 up, 0 in 2024-02-14T17:15:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: mgrmap e13: y(active, since 47s) 2024-02-14T17:15:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-14T17:15:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: Cluster is now healthy 2024-02-14T17:15:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: overall HEALTH_OK 2024-02-14T17:15:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:15:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:53 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:55.166 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[24769]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:15:55.167 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[24769]: Reconfiguring daemon mon.c on smithi017 2024-02-14T17:15:55.167 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:55.167 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[24769]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:55.167 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[29506]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:15:55.167 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[29506]: Reconfiguring daemon mon.c on smithi017 2024-02-14T17:15:55.167 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:55.167 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:54 smithi017 ceph-mon[29506]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:55.197 INFO:teuthology.orchestra.run.smithi017.stdout:# minimal ceph.conf for 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:15:55.197 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2024-02-14T17:15:55.198 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:15:55.198 INFO:teuthology.orchestra.run.smithi017.stdout: mon_host = [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] 2024-02-14T17:15:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:54 smithi111 ceph-mon[25603]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:15:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:54 smithi111 ceph-mon[25603]: Reconfiguring daemon mon.c on smithi017 2024-02-14T17:15:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:54 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:15:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:54 smithi111 ceph-mon[25603]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:55.899 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-14T17:15:55.900 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:15:55.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.conf 2024-02-14T17:15:55.929 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:15:55.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:55.996 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-14T17:15:55.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-02-14T17:15:56.029 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-14T17:15:56.029 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:15:56.100 INFO:tasks.cephadm:Adding mgr.y on smithi017 2024-02-14T17:15:56.100 INFO:tasks.cephadm:Adding mgr.x on smithi111 2024-02-14T17:15:56.100 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch apply mgr '2;smithi017=y;smithi111=x' 2024-02-14T17:15:56.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:55 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1030225289' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:55 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1030225289' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:55 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1030225289' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:57.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:57 smithi111 ceph-mon[25603]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:57.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:57.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:57 smithi111 ceph-mon[25603]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:15:57.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:57.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:15:57.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:57.482 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:57 smithi111 ceph-mon[25603]: Reconfiguring daemon mon.b on smithi111 2024-02-14T17:15:57.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[24769]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[24769]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[24769]: Reconfiguring daemon mon.b on smithi111 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[29506]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[29506]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:15:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:15:57.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:57 smithi017 ceph-mon[29506]: Reconfiguring daemon mon.b on smithi111 2024-02-14T17:15:58.267 INFO:teuthology.orchestra.run.smithi111.stdout:Scheduled mgr update... 2024-02-14T17:15:58.952 DEBUG:teuthology.orchestra.run.smithi111:mgr.x> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service 2024-02-14T17:15:58.955 INFO:tasks.cephadm:Deploying OSDs... 2024-02-14T17:15:58.955 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:15:58.955 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:15:58.974 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-14T17:15:58.974 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-02-14 17:15:35.424367786 +0000 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-02-14 17:13:06.918807976 +0000 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-02-14 17:13:06.918807976 +0000 2024-02-14T17:15:59.032 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-02-14 17:13:06.918807976 +0000 2024-02-14T17:15:59.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-14T17:15:59.101 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-02-14T17:15:59.101 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-02-14T17:15:59.101 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000119262 s, 4.3 MB/s 2024-02-14T17:15:59.103 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-14T17:15:59.161 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-02-14T17:15:59.220 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-14T17:15:59.220 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:15:59.220 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-14T17:15:59.220 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:15:59.220 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:15:59.220 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-02-14 17:15:35.424367786 +0000 2024-02-14T17:15:59.221 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-02-14 17:13:07.261803937 +0000 2024-02-14T17:15:59.221 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-02-14 17:13:07.261803937 +0000 2024-02-14T17:15:59.221 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-02-14 17:13:07.261803937 +0000 2024-02-14T17:15:59.221 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-14T17:15:59.289 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-02-14T17:15:59.289 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-02-14T17:15:59.289 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000147748 s, 3.5 MB/s 2024-02-14T17:15:59.291 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-14T17:15:59.348 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-02-14 17:15:35.425367774 +0000 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-02-14 17:13:07.605799886 +0000 2024-02-14T17:15:59.404 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-02-14 17:13:07.605799886 +0000 2024-02-14T17:15:59.405 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-02-14 17:13:07.605799886 +0000 2024-02-14T17:15:59.405 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-14T17:15:59.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[24769]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:59.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[24769]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi111=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[24769]: Saving service mgr spec with placement smithi017=y;smithi111=x;count:2 2024-02-14T17:15:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[29506]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[29506]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi111=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[29506]: Saving service mgr spec with placement smithi017=y;smithi111=x;count:2 2024-02-14T17:15:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:15:59 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:59.466 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-02-14T17:15:59.466 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-02-14T17:15:59.466 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000109771 s, 4.7 MB/s 2024-02-14T17:15:59.467 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-14T17:15:59.523 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-02-14T17:15:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-14T17:15:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:15:59.580 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-14T17:15:59.580 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:15:59.580 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:15:59.580 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-02-14 17:15:35.425367774 +0000 2024-02-14T17:15:59.580 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-02-14 17:13:07.940795941 +0000 2024-02-14T17:15:59.580 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-02-14 17:13:07.940795941 +0000 2024-02-14T17:15:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-02-14 17:13:07.940795941 +0000 2024-02-14T17:15:59.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-14T17:15:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-02-14T17:15:59.645 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-02-14T17:15:59.646 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000108756 s, 4.7 MB/s 2024-02-14T17:15:59.647 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-14T17:15:59.703 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-14T17:15:59.703 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-02-14T17:15:59.719 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-14T17:15:59.719 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-02-14T17:15:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:59 smithi111 ceph-mon[25603]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:15:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:59 smithi111 ceph-mon[25603]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi017=y;smithi111=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:15:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:59 smithi111 ceph-mon[25603]: Saving service mgr spec with placement smithi017=y;smithi111=x;count:2 2024-02-14T17:15:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:15:59 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-02-14 17:15:44.650841420 +0000 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-02-14 17:13:04.128579926 +0000 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-02-14 17:13:04.128579926 +0000 2024-02-14T17:15:59.749 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-02-14 17:13:04.128579926 +0000 2024-02-14T17:15:59.749 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-14T17:15:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-02-14T17:15:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-02-14T17:15:59.815 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000207128 s, 2.5 MB/s 2024-02-14T17:15:59.816 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-14T17:15:59.873 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-02-14 17:15:44.651841402 +0000 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-02-14 17:13:04.455573915 +0000 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-02-14 17:13:04.455573915 +0000 2024-02-14T17:15:59.928 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-02-14 17:13:04.455573915 +0000 2024-02-14T17:15:59.929 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-14T17:15:59.993 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-02-14T17:15:59.993 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-02-14T17:15:59.994 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000227069 s, 2.3 MB/s 2024-02-14T17:15:59.995 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-14T17:16:00.051 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-02-14T17:16:00.105 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-14T17:16:00.106 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:16:00.106 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-14T17:16:00.106 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:16:00.106 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:16:00.106 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-02-14 17:15:44.651841402 +0000 2024-02-14T17:16:00.106 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-02-14 17:13:04.781567923 +0000 2024-02-14T17:16:00.107 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-02-14 17:13:04.781567923 +0000 2024-02-14T17:16:00.107 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-02-14 17:13:04.781567923 +0000 2024-02-14T17:16:00.107 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-14T17:16:00.171 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-02-14T17:16:00.172 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-02-14T17:16:00.172 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000223425 s, 2.3 MB/s 2024-02-14T17:16:00.173 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-14T17:16:00.228 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-02-14 17:15:44.651841402 +0000 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-02-14 17:13:05.265559025 +0000 2024-02-14T17:16:00.283 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-02-14 17:13:05.265559025 +0000 2024-02-14T17:16:00.284 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-02-14 17:13:05.265559025 +0000 2024-02-14T17:16:00.284 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-14T17:16:00.348 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-02-14T17:16:00.348 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-02-14T17:16:00.348 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000193179 s, 2.7 MB/s 2024-02-14T17:16:00.349 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-14T17:16:00.406 INFO:tasks.cephadm:Deploying osd.0 on smithi017 with /dev/vg_nvme/lv_4... 2024-02-14T17:16:00.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[29506]: Deploying daemon mgr.x on smithi111 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:00 smithi017 ceph-mon[24769]: Deploying daemon mgr.x on smithi111 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-14T17:16:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:16:00.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:00.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:00 smithi111 ceph-mon[25603]: Deploying daemon mgr.x on smithi111 2024-02-14T17:16:01.362 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:01 smithi111 podman[26766]: 2024-02-14 17:16:01.285898583 +0000 UTC m=+0.114562701 container create d90044f2e9a43bea02916ba67e058d1aeed159403b26351cb8afb8efa6148149 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.tags=base centos centos-stream, version=8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.19.8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:16:01.628 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:01 smithi017 ceph-mon[24769]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:01.628 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:01 smithi017 ceph-mon[29506]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:01.637 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:01 smithi111 ceph-mon[25603]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:01.637 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:01 smithi111 podman[26766]: 2024-02-14 17:16:01.479927771 +0000 UTC m=+0.308591888 container init d90044f2e9a43bea02916ba67e058d1aeed159403b26351cb8afb8efa6148149 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=) 2024-02-14T17:16:01.637 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:01 smithi111 podman[26766]: 2024-02-14 17:16:01.482446714 +0000 UTC m=+0.311110839 container start d90044f2e9a43bea02916ba67e058d1aeed159403b26351cb8afb8efa6148149 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=centos-stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vcs-type=git, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, ceph=True, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:16:01.637 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:01 smithi111 ceph-mgr[26785]: -- 172.21.15.111:0/1643406019 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c15550340 con 0x561c15528800 2024-02-14T17:16:01.637 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:01 smithi111 bash[26766]: d90044f2e9a43bea02916ba67e058d1aeed159403b26351cb8afb8efa6148149 2024-02-14T17:16:01.637 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:01 smithi111 systemd[1]: Started Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:16:02.325 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:01 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:01.989+0000 7f909a096000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:16:02.325 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:02.081+0000 7f909a096000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:16:02.334 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-14T17:16:02.334 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-14T17:16:02.334 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-02-14T17:16:02.334 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-02-14T17:16:02.334 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00832325 s, 1.3 GB/s 2024-02-14T17:16:02.334 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:16:02.916 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:16:02.917 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:02.918 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_4 2024-02-14T17:16:02.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:02.689+0000 7f909a096000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:16:02.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:02.776+0000 7f909a096000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:16:02.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:02.857+0000 7f909a096000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:16:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:16:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:03.274 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:03.003+0000 7f909a096000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:16:03.275 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:03.083+0000 7f909a096000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:16:03.275 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:03.191+0000 7f909a096000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:16:03.275 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:03.273+0000 7f909a096000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:16:03.647 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:03.434+0000 7f909a096000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:16:03.647 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:03.520+0000 7f909a096000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:16:03.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:03 smithi017 ceph-mon[24769]: Reconfiguring mgr.y (unknown last config time)... 2024-02-14T17:16:03.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:03 smithi017 ceph-mon[24769]: Reconfiguring daemon mgr.y on smithi017 2024-02-14T17:16:03.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:03 smithi017 ceph-mon[29506]: Reconfiguring mgr.y (unknown last config time)... 2024-02-14T17:16:03.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:03 smithi017 ceph-mon[29506]: Reconfiguring daemon mgr.y on smithi017 2024-02-14T17:16:03.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-mon[25603]: Reconfiguring mgr.y (unknown last config time)... 2024-02-14T17:16:03.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:03 smithi111 ceph-mon[25603]: Reconfiguring daemon mgr.y on smithi017 2024-02-14T17:16:04.865 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:04.548+0000 7f909a096000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:16:04.865 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:04.677+0000 7f909a096000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:16:04.865 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:04.863+0000 7f909a096000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:16:05.130 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[24769]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:05.130 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:05.130 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:05.130 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:05.130 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:05.130 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:05.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[29506]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:05.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:05.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:05.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:05.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:05.131 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:04.950+0000 7f909a096000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:05.086+0000 7f909a096000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-mon[25603]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:05.705 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:05.400+0000 7f909a096000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:16:05.705 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:05.619+0000 7f909a096000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:16:05.705 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:05.704+0000 7f909a096000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:16:05.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:05.871+0000 7f909a096000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:16:06.310 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:05 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:06.311 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:05 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:06.311 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:05 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:06.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:05 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:06.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:05 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:06.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:05 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:05 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:07.076 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:06 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:06.784+0000 7f909a096000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:16:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:06 smithi111 ceph-mon[25603]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:06 smithi111 ceph-mon[25603]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:16:07.077 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:16:07.078 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[24769]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[24769]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[29506]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[29506]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:16:07.165 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:16:07.166 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:07.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:07.075+0000 7f909a096000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:16:07.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:07.168+0000 7f909a096000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:16:07.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:16:07.317+0000 7f909a096000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: Standby manager daemon x started 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/658166525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a980d777-c667-4148-b5fa-b9fc89d6dc45"}]: dispatch 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/658166525' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a980d777-c667-4148-b5fa-b9fc89d6dc45"}]': finished 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: osdmap e5: 1 total, 0 up, 1 in 2024-02-14T17:16:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/658166525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a980d777-c667-4148-b5fa-b9fc89d6dc45"}]: dispatch 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/658166525' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a980d777-c667-4148-b5fa-b9fc89d6dc45"}]': finished 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: osdmap e5: 1 total, 0 up, 1 in 2024-02-14T17:16:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:07 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: Standby manager daemon x started 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3614867820' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/658166525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a980d777-c667-4148-b5fa-b9fc89d6dc45"}]: dispatch 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/658166525' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a980d777-c667-4148-b5fa-b9fc89d6dc45"}]': finished 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: osdmap e5: 1 total, 0 up, 1 in 2024-02-14T17:16:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:07 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:09.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[29506]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[29506]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-14T17:16:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:16:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/412723927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[24769]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[24769]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-14T17:16:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:16:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:09 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/412723927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:09 smithi111 ceph-mon[25603]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:09 smithi111 ceph-mon[25603]: mgrmap e14: y(active, since 62s), standbys: x 2024-02-14T17:16:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:09 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:16:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:09 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/412723927' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:10 smithi111 ceph-mon[25603]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:11.295 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:10 smithi017 ceph-mon[24769]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:11.295 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:10 smithi017 ceph-mon[29506]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:12 smithi111 ceph-mon[25603]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:13.261 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:12 smithi017 ceph-mon[24769]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:13.262 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:12 smithi017 ceph-mon[29506]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[24769]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:16:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[24769]: Deploying daemon osd.0 on smithi017 2024-02-14T17:16:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[29506]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:16:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:15.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:14 smithi017 ceph-mon[29506]: Deploying daemon osd.0 on smithi017 2024-02-14T17:16:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:14 smithi111 ceph-mon[25603]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:16:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:14 smithi111 ceph-mon[25603]: Deploying daemon osd.0 on smithi017 2024-02-14T17:16:17.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:16 smithi017 ceph-mon[24769]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:17.127 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:16 smithi017 ceph-mon[29506]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:17.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:16 smithi111 ceph-mon[25603]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:18 smithi111 ceph-mon[25603]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:18 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:18 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:18 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:18 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:19.273 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[29506]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:19.273 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:19.273 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:19.273 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:19.273 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:19.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[24769]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:19.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:19.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:19.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:19.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:18 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:21.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:20 smithi111 ceph-mon[25603]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:21.284 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:20 smithi017 ceph-mon[24769]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:21.285 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:20 smithi017 ceph-mon[29506]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:21.531 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 0 on host 'smithi017' 2024-02-14T17:16:22.114 DEBUG:teuthology.orchestra.run.smithi017:osd.0> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.0.service 2024-02-14T17:16:22.117 INFO:tasks.cephadm:Deploying osd.1 on smithi017 with /dev/vg_nvme/lv_3... 2024-02-14T17:16:22.117 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-14T17:16:22.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:22.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[24769]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:16:22.722 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[24769]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:22.723 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:22.723 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:22.723 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[29506]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:16:22.723 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[29506]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:22.723 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:22 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:22 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:22 smithi111 ceph-mon[25603]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:16:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:22 smithi111 ceph-mon[25603]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:22 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[24769]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[24769]: osdmap e6: 1 total, 0 up, 1 in 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[24769]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[29506]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[29506]: osdmap e6: 1 total, 0 up, 1 in 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:23.619 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-mon[29506]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:16:23.620 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[32363]: 2024-02-14T17:16:23.540+0000 7f41e36c3700 -1 osd.0 0 waiting for initial osdmap 2024-02-14T17:16:23.620 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:16:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[32363]: 2024-02-14T17:16:23.544+0000 7f41e005e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:16:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:23 smithi111 ceph-mon[25603]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:16:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:23 smithi111 ceph-mon[25603]: osdmap e6: 1 total, 0 up, 1 in 2024-02-14T17:16:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:23 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:23 smithi111 ceph-mon[25603]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:16:24.104 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-14T17:16:24.104 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-14T17:16:24.104 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-02-14T17:16:24.105 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-02-14T17:16:24.105 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00954461 s, 1.1 GB/s 2024-02-14T17:16:24.105 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-02-14T17:16:24.855 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[29506]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:16:24.855 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[29506]: osdmap e7: 1 total, 0 up, 1 in 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[29506]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[24769]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[24769]: osdmap e7: 1 total, 0 up, 1 in 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:24.856 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:24 smithi017 ceph-mon[24769]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:24 smithi111 ceph-mon[25603]: from='osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:16:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:24 smithi111 ceph-mon[25603]: osdmap e7: 1 total, 0 up, 1 in 2024-02-14T17:16:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:24 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:24 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:24 smithi111 ceph-mon[25603]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-14T17:16:25.786 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:16:25.786 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:16:25.786 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:25.786 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[24769]: osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272] boot 2024-02-14T17:16:25.786 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[24769]: osdmap e8: 1 total, 1 up, 1 in 2024-02-14T17:16:25.786 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:25.786 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:16:25.787 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:16:25.787 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:25.787 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[29506]: osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272] boot 2024-02-14T17:16:25.787 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[29506]: osdmap e8: 1 total, 1 up, 1 in 2024-02-14T17:16:25.787 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:25 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:25.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_3 2024-02-14T17:16:25.876 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:25 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:16:25.876 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:25 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:16:25.876 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:25 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:25.876 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:25 smithi111 ceph-mon[25603]: osd.0 [v2:172.21.15.17:6802/2126863272,v1:172.21.15.17:6803/2126863272] boot 2024-02-14T17:16:25.876 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:25 smithi111 ceph-mon[25603]: osdmap e8: 1 total, 1 up, 1 in 2024-02-14T17:16:25.876 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:25 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:16:26.710 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:26 smithi111 ceph-mon[25603]: osdmap e9: 1 total, 1 up, 1 in 2024-02-14T17:16:26.710 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:26 smithi111 ceph-mon[25603]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:26.823 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:26 smithi017 ceph-mon[24769]: osdmap e9: 1 total, 1 up, 1 in 2024-02-14T17:16:26.824 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:26 smithi017 ceph-mon[24769]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:26.824 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:26 smithi017 ceph-mon[29506]: osdmap e9: 1 total, 1 up, 1 in 2024-02-14T17:16:26.824 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:26 smithi017 ceph-mon[29506]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:27.899 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi017 2024-02-14T17:16:27.899 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.899 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:27.899 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi017 to 16069M 2024-02-14T17:16:27.899 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi017 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi017 to 16069M 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.900 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.901 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:27.901 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:27.901 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:27.901 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi017 2024-02-14T17:16:27.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:27.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi017 to 16069M 2024-02-14T17:16:27.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:27.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:27.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:27.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:27.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[29506]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[29506]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[24769]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[24769]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:28.885 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:28.886 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:28.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:28 smithi111 ceph-mon[25603]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:28.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:28 smithi111 ceph-mon[25603]: from='client.24142 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:28.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:28 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:28.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:28 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:28.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:28 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:30 smithi111 ceph-mon[25603]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3082859409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "517877bd-896e-4e1d-93e6-2a704bca2e77"}]: dispatch 2024-02-14T17:16:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3082859409' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "517877bd-896e-4e1d-93e6-2a704bca2e77"}]': finished 2024-02-14T17:16:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:30 smithi111 ceph-mon[25603]: osdmap e10: 2 total, 1 up, 2 in 2024-02-14T17:16:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:30 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[24769]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3082859409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "517877bd-896e-4e1d-93e6-2a704bca2e77"}]: dispatch 2024-02-14T17:16:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3082859409' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "517877bd-896e-4e1d-93e6-2a704bca2e77"}]': finished 2024-02-14T17:16:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[24769]: osdmap e10: 2 total, 1 up, 2 in 2024-02-14T17:16:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:31.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[29506]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:31.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3082859409' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "517877bd-896e-4e1d-93e6-2a704bca2e77"}]: dispatch 2024-02-14T17:16:31.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3082859409' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "517877bd-896e-4e1d-93e6-2a704bca2e77"}]': finished 2024-02-14T17:16:31.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[29506]: osdmap e10: 2 total, 1 up, 2 in 2024-02-14T17:16:31.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:30 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:32.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:31 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2223374777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:32.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:31 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2223374777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:31 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2223374777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:32 smithi111 ceph-mon[25603]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:33.299 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:32 smithi017 ceph-mon[24769]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:33.300 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:32 smithi017 ceph-mon[29506]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:35.170 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:34 smithi017 ceph-mon[29506]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:35.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:34 smithi017 ceph-mon[24769]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:34 smithi111 ceph-mon[25603]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:37.101 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[24769]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:37.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:16:37.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:37.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[24769]: Deploying daemon osd.1 on smithi017 2024-02-14T17:16:37.102 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[29506]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:37.102 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:16:37.102 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:37.102 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:36 smithi017 ceph-mon[29506]: Deploying daemon osd.1 on smithi017 2024-02-14T17:16:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:36 smithi111 ceph-mon[25603]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:16:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:36 smithi111 ceph-mon[25603]: Deploying daemon osd.1 on smithi017 2024-02-14T17:16:39.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:38 smithi017 ceph-mon[24769]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:39.186 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:38 smithi017 ceph-mon[29506]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:38 smithi111 ceph-mon[25603]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:41.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:40 smithi017 ceph-mon[24769]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:41.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:40 smithi017 ceph-mon[29506]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:41.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:40 smithi111 ceph-mon[25603]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:42.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:42.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:42.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:43.386 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:43 smithi017 ceph-mon[24769]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:43.386 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:43 smithi017 ceph-mon[29506]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:43.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:43 smithi111 ceph-mon[25603]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:44.314 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 1 on host 'smithi017' 2024-02-14T17:16:44.815 DEBUG:teuthology.orchestra.run.smithi017:osd.1> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.1.service 2024-02-14T17:16:44.817 INFO:tasks.cephadm:Deploying osd.2 on smithi017 with /dev/vg_nvme/lv_2... 2024-02-14T17:16:44.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-14T17:16:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:45 smithi111 ceph-mon[25603]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:45 smithi111 ceph-mon[25603]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:16:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:45 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:45 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:45.499 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[29506]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:45.499 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[29506]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:16:45.500 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:45.500 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:45.500 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[24769]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:45.500 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[24769]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:16:45.500 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:45.500 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:45 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:46.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[24769]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:16:46.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[24769]: osdmap e11: 2 total, 1 up, 2 in 2024-02-14T17:16:46.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:46.306 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[24769]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:16:46.306 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[29506]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:16:46.306 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[29506]: osdmap e11: 2 total, 1 up, 2 in 2024-02-14T17:16:46.306 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:46.306 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-mon[29506]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:16:46.306 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[35320]: 2024-02-14T17:16:46.227+0000 7f490d2d8700 -1 osd.1 0 waiting for initial osdmap 2024-02-14T17:16:46.306 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:16:46 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[35320]: 2024-02-14T17:16:46.230+0000 7f4907c6f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:16:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:46 smithi111 ceph-mon[25603]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:16:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:46 smithi111 ceph-mon[25603]: osdmap e11: 2 total, 1 up, 2 in 2024-02-14T17:16:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:46 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:46 smithi111 ceph-mon[25603]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:16:46.679 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-14T17:16:46.679 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-14T17:16:46.679 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-02-14T17:16:46.679 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-02-14T17:16:46.679 INFO:teuthology.orchestra.run.smithi017.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00978553 s, 1.1 GB/s 2024-02-14T17:16:46.679 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-02-14T17:16:47.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[24769]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:47.349 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[24769]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[24769]: osdmap e12: 2 total, 1 up, 2 in 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[29506]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[29506]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[29506]: osdmap e12: 2 total, 1 up, 2 in 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:47.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:47 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:47.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:47 smithi111 ceph-mon[25603]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-02-14T17:16:47.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:47 smithi111 ceph-mon[25603]: from='osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:16:47.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:47 smithi111 ceph-mon[25603]: osdmap e12: 2 total, 1 up, 2 in 2024-02-14T17:16:47.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:47 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:47.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:47 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[24769]: osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104] boot 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[24769]: osdmap e13: 2 total, 2 up, 2 in 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:16:48.436 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[29506]: osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104] boot 2024-02-14T17:16:48.437 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[29506]: osdmap e13: 2 total, 2 up, 2 in 2024-02-14T17:16:48.437 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:48 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:48.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_2 2024-02-14T17:16:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:48 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:16:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:48 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:16:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:48 smithi111 ceph-mon[25603]: osd.1 [v2:172.21.15.17:6810/336965104,v1:172.21.15.17:6811/336965104] boot 2024-02-14T17:16:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:48 smithi111 ceph-mon[25603]: osdmap e13: 2 total, 2 up, 2 in 2024-02-14T17:16:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:48 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[24769]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[24769]: osdmap e14: 2 total, 2 up, 2 in 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[29506]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[29506]: osdmap e14: 2 total, 2 up, 2 in 2024-02-14T17:16:49.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:49.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:49.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:49.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:49 smithi111 ceph-mon[25603]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:49 smithi111 ceph-mon[25603]: osdmap e14: 2 total, 2 up, 2 in 2024-02-14T17:16:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:16:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:50.302 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi017 2024-02-14T17:16:50.302 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi017 to 8034M 2024-02-14T17:16:50.302 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:50.302 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:50.302 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:50.303 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:50.303 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:50.303 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi017 2024-02-14T17:16:50.303 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi017 to 8034M 2024-02-14T17:16:50.303 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:50.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:50.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:50.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:50.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:50 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:50 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi017 2024-02-14T17:16:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:50 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi017 to 8034M 2024-02-14T17:16:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:50 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:50 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:16:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:50 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:50 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:16:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:50 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:16:51.325 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[29506]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:51.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:51.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:51.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:51.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[24769]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:51.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:51.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:51.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:51 smithi111 ceph-mon[25603]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:51 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:16:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:51 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:16:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:51 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:52 smithi017 ceph-mon[24769]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:52 smithi017 ceph-mon[29506]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:52 smithi111 ceph-mon[25603]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[24769]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1457758430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c810adb5-4205-4e6a-a102-8cdeafeb7964"}]: dispatch 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1457758430' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c810adb5-4205-4e6a-a102-8cdeafeb7964"}]': finished 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[24769]: osdmap e15: 3 total, 2 up, 3 in 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[29506]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1457758430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c810adb5-4205-4e6a-a102-8cdeafeb7964"}]: dispatch 2024-02-14T17:16:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1457758430' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c810adb5-4205-4e6a-a102-8cdeafeb7964"}]': finished 2024-02-14T17:16:53.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[29506]: osdmap e15: 3 total, 2 up, 3 in 2024-02-14T17:16:53.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:53 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:16:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:53 smithi111 ceph-mon[25603]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:53 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1457758430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c810adb5-4205-4e6a-a102-8cdeafeb7964"}]: dispatch 2024-02-14T17:16:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:53 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1457758430' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c810adb5-4205-4e6a-a102-8cdeafeb7964"}]': finished 2024-02-14T17:16:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:53 smithi111 ceph-mon[25603]: osdmap e15: 3 total, 2 up, 3 in 2024-02-14T17:16:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:53 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:16:54.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:54 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1351250062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:54 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1351250062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:54 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1351250062' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:16:55.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:55 smithi017 ceph-mon[24769]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:55.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:55 smithi017 ceph-mon[29506]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:55 smithi111 ceph-mon[25603]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:57.534 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:57 smithi017 ceph-mon[24769]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:57.534 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:57 smithi017 ceph-mon[29506]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:57.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:57 smithi111 ceph-mon[25603]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:59.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:59 smithi017 ceph-mon[24769]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:59.586 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:59 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:16:59.586 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:16:59 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:59.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:59 smithi017 ceph-mon[29506]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:59.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:59 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:16:59.587 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:16:59 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:16:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:59 smithi111 ceph-mon[25603]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:16:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:59 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:16:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:16:59 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:00.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:00 smithi017 ceph-mon[24769]: Deploying daemon osd.2 on smithi017 2024-02-14T17:17:00.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:00 smithi017 ceph-mon[29506]: Deploying daemon osd.2 on smithi017 2024-02-14T17:17:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:00 smithi111 ceph-mon[25603]: Deploying daemon osd.2 on smithi017 2024-02-14T17:17:01.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:01 smithi017 ceph-mon[24769]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:01.468 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:01 smithi017 ceph-mon[29506]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:01 smithi111 ceph-mon[25603]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:03.268 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:03 smithi017 ceph-mon[24769]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:03.541 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:03 smithi017 ceph-mon[29506]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:03 smithi111 ceph-mon[25603]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:05.118 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:05.118 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[29506]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:05.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:04 smithi017 ceph-mon[24769]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:04 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:04 smithi111 ceph-mon[25603]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:06.868 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 2 on host 'smithi017' 2024-02-14T17:17:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[29506]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:17:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:17:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[29506]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:17:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[24769]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:17:07.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:17:07.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:07.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:06 smithi017 ceph-mon[24769]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:17:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:06 smithi111 ceph-mon[25603]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:17:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:17:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:06 smithi111 ceph-mon[25603]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:17:07.379 DEBUG:teuthology.orchestra.run.smithi017:osd.2> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.2.service 2024-02-14T17:17:07.381 INFO:tasks.cephadm:Deploying osd.3 on smithi017 with /dev/vg_nvme/lv_1... 2024-02-14T17:17:07.382 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-14T17:17:08.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[29506]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[29506]: osdmap e16: 3 total, 2 up, 3 in 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[29506]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[29506]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[24769]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[24769]: osdmap e16: 3 total, 2 up, 3 in 2024-02-14T17:17:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[24769]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:17:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-mon[24769]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:08 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:08 smithi111 ceph-mon[25603]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:17:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:08 smithi111 ceph-mon[25603]: osdmap e16: 3 total, 2 up, 3 in 2024-02-14T17:17:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:08 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:08 smithi111 ceph-mon[25603]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:17:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:08 smithi111 ceph-mon[25603]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:09.006 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[38229]: 2024-02-14T17:17:08.870+0000 7f70d785a700 -1 osd.2 0 waiting for initial osdmap 2024-02-14T17:17:09.006 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:08 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[38229]: 2024-02-14T17:17:08.874+0000 7f70d19f0700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:17:10.072 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[24769]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:17:10.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[24769]: osdmap e17: 3 total, 2 up, 3 in 2024-02-14T17:17:10.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:10.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:10.073 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[29506]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:17:10.073 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[29506]: osdmap e17: 3 total, 2 up, 3 in 2024-02-14T17:17:10.073 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:10.073 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:09 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:10.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:09 smithi111 ceph-mon[25603]: from='osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:17:10.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:09 smithi111 ceph-mon[25603]: osdmap e17: 3 total, 2 up, 3 in 2024-02-14T17:17:10.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:09 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:10.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:09 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:10.544 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-14T17:17:10.545 INFO:teuthology.orchestra.run.smithi017.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-14T17:17:10.545 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10+0 records in 2024-02-14T17:17:10.545 INFO:teuthology.orchestra.run.smithi017.stderr:10+0 records out 2024-02-14T17:17:10.545 INFO:teuthology.orchestra.run.smithi017.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108902 s, 963 MB/s 2024-02-14T17:17:10.545 INFO:teuthology.orchestra.run.smithi017.stderr:--> Zapping successful for: 2024-02-14T17:17:11.178 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:17:11.178 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:17:11.178 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[24769]: osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458] boot 2024-02-14T17:17:11.178 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[24769]: osdmap e18: 3 total, 3 up, 3 in 2024-02-14T17:17:11.178 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:11.179 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[24769]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:11.179 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-14T17:17:11.179 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:17:11.179 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:17:11.180 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[29506]: osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458] boot 2024-02-14T17:17:11.180 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[29506]: osdmap e18: 3 total, 3 up, 3 in 2024-02-14T17:17:11.180 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:11.180 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[29506]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:11.180 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:10 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-14T17:17:11.201 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi017:vg_nvme/lv_1 2024-02-14T17:17:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:10 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:17:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:10 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:17:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:10 smithi111 ceph-mon[25603]: osd.2 [v2:172.21.15.17:6818/3232569458,v1:172.21.15.17:6819/3232569458] boot 2024-02-14T17:17:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:10 smithi111 ceph-mon[25603]: osdmap e18: 3 total, 3 up, 3 in 2024-02-14T17:17:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:10 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:17:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:10 smithi111 ceph-mon[25603]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-02-14T17:17:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:10 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: osdmap e19: 3 total, 3 up, 3 in 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.053 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:12.054 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:12.054 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:12.054 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.054 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-14T17:17:12.054 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: osdmap e19: 3 total, 3 up, 3 in 2024-02-14T17:17:12.054 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:12.055 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-14T17:17:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: osdmap e19: 3 total, 3 up, 3 in 2024-02-14T17:17:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-14T17:17:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:12.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:12.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:12.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:12.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:12.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:13.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi017 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi017 to 5356M 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[24769]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[24769]: osdmap e20: 3 total, 3 up, 3 in 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi017 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi017 to 5356M 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[29506]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:13.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-14T17:17:13.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:12 smithi017 ceph-mon[29506]: osdmap e20: 3 total, 3 up, 3 in 2024-02-14T17:17:13.191 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39537]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:17:13.191 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39537]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:13.191 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39537]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:12 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi017 2024-02-14T17:17:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:12 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi017 to 5356M 2024-02-14T17:17:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:12 smithi111 ceph-mon[25603]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:12 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-14T17:17:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:12 smithi111 ceph-mon[25603]: osdmap e20: 3 total, 3 up, 3 in 2024-02-14T17:17:13.640 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39546]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:17:13.640 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39546]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:13.640 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39546]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:13.907 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39587]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:17:13.907 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39587]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:13.907 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39587]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[24769]: osdmap e21: 3 total, 3 up, 3 in 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[29506]: osdmap e21: 3 total, 3 up, 3 in 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:14.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:14.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:14.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:13 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:14.169 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39649]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:17:14.169 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39649]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:14.169 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:17:13 smithi017 sudo[39649]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:13 smithi111 ceph-mon[25603]: osdmap e21: 3 total, 3 up, 3 in 2024-02-14T17:17:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:13 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:13 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:13 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:13 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:14.437 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39660]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:17:14.437 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39660]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:14.437 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39660]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:14.689 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39676]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:17:14.689 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39676]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:14.689 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39676]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:14.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39683]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:17:14.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39683]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:14.959 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39683]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:15.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39705]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:17:15.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39705]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:15.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:14 smithi017 sudo[39705]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:15.488 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 sudo[39714]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:17:15.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 sudo[39714]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:15.489 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 sudo[39714]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:15.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 sudo[39726]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:17:15.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 sudo[39726]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:15.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 sudo[39726]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:15.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 sudo[27102]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-14T17:17:15.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 sudo[27102]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:15.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 sudo[27102]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: pgmap v75: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: pgmap v75: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 sudo[27105]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-14T17:17:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 sudo[27105]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-14T17:17:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 sudo[27105]: pam_unix(sudo:session): session closed for user root 2024-02-14T17:17:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi017:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: pgmap v75: 1 pgs: 1 creating+activating; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-14T17:17:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/181216421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]: dispatch 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]: dispatch 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]': finished 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: osdmap e22: 4 total, 3 up, 4 in 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4232120198' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/181216421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]: dispatch 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]: dispatch 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]': finished 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: osdmap e22: 4 total, 3 up, 4 in 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:17.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:16 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4232120198' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:17:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-14T17:17:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-14T17:17:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:17:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/181216421' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]: dispatch 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]: dispatch 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c10a4f3-4cd7-44da-8859-96c78d7b5da0"}]': finished 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: osdmap e22: 4 total, 3 up, 4 in 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:17.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:16 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4232120198' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:17:19.179 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:18 smithi017 ceph-mon[24769]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:19.179 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:18 smithi017 ceph-mon[29506]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:18 smithi111 ceph-mon[25603]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:20 smithi017 ceph-mon[24769]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:20 smithi017 ceph-mon[29506]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:20 smithi111 ceph-mon[25603]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[24769]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[24769]: Deploying daemon osd.3 on smithi017 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[29506]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:22 smithi017 ceph-mon[29506]: Deploying daemon osd.3 on smithi017 2024-02-14T17:17:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:22 smithi111 ceph-mon[25603]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:22 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:17:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:22 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:22 smithi111 ceph-mon[25603]: Deploying daemon osd.3 on smithi017 2024-02-14T17:17:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:25 smithi017 ceph-mon[24769]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:25 smithi017 ceph-mon[29506]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:25.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:25 smithi111 ceph-mon[25603]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:27.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[24769]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[29506]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:27.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:27 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:27.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:27 smithi111 ceph-mon[25603]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:27.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:27.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:27.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:27.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:27 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:29.397 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:29 smithi017 ceph-mon[24769]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:29.397 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:29 smithi017 ceph-mon[29506]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:29 smithi111 ceph-mon[25603]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:30.055 INFO:teuthology.orchestra.run.smithi017.stdout:Created osd(s) 3 on host 'smithi017' 2024-02-14T17:17:30.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:30.617 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:17:30.617 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.617 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:30.617 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:30.617 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:30 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:30.691 DEBUG:teuthology.orchestra.run.smithi017:osd.3> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.3.service 2024-02-14T17:17:30.694 INFO:tasks.cephadm:Deploying osd.4 on smithi111 with /dev/vg_nvme/lv_4... 2024-02-14T17:17:30.694 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-14T17:17:30.718 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.718 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.718 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:30.718 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:17:30.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:30.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:30.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:30.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:30 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:31.976 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[24769]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-14T17:17:32.199 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:31 smithi111 ceph-mon[25603]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-14T17:17:32.200 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:31 smithi111 ceph-mon[25603]: osdmap e23: 4 total, 3 up, 4 in 2024-02-14T17:17:32.200 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:31 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:32.200 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:31 smithi111 ceph-mon[25603]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:17:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[29506]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-14T17:17:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[29506]: osdmap e23: 4 total, 3 up, 4 in 2024-02-14T17:17:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:32.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[29506]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:17:32.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[24769]: osdmap e23: 4 total, 3 up, 4 in 2024-02-14T17:17:32.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:32.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-mon[24769]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:17:32.443 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[41194]: 2024-02-14T17:17:31.977+0000 7f04daa47700 -1 osd.3 0 waiting for initial osdmap 2024-02-14T17:17:32.443 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:17:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[41194]: 2024-02-14T17:17:31.982+0000 7f04d6be1700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:17:32.695 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-14T17:17:32.695 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-14T17:17:32.696 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-02-14T17:17:32.696 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-02-14T17:17:32.696 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110966 s, 945 MB/s 2024-02-14T17:17:32.696 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-02-14T17:17:33.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[24769]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:33.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[24769]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:17:33.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[24769]: osdmap e24: 4 total, 3 up, 4 in 2024-02-14T17:17:33.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:33.082 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:33.082 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[29506]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:33.082 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[29506]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:17:33.083 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[29506]: osdmap e24: 4 total, 3 up, 4 in 2024-02-14T17:17:33.083 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:33.083 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:32 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:32 smithi111 ceph-mon[25603]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:32 smithi111 ceph-mon[25603]: from='osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]': finished 2024-02-14T17:17:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:32 smithi111 ceph-mon[25603]: osdmap e24: 4 total, 3 up, 4 in 2024-02-14T17:17:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:32 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:32 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:33.287 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-02-14T17:17:34.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:17:34.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:17:34.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823] boot 2024-02-14T17:17:34.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: osdmap e25: 4 total, 4 up, 4 in 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi017 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi017 to 4017M 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823] boot 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: osdmap e25: 4 total, 4 up, 4 in 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi017 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi017 to 4017M 2024-02-14T17:17:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: osd.3 [v2:172.21.15.17:6826/2789642823,v1:172.21.15.17:6827/2789642823] boot 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: osdmap e25: 4 total, 4 up, 4 in 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi017 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:34.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi017 to 4017M 2024-02-14T17:17:34.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:34.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:35.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:34 smithi111 ceph-mon[25603]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:35.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:34 smithi111 ceph-mon[25603]: osdmap e26: 4 total, 4 up, 4 in 2024-02-14T17:17:35.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:34 smithi017 ceph-mon[24769]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:35.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:34 smithi017 ceph-mon[24769]: osdmap e26: 4 total, 4 up, 4 in 2024-02-14T17:17:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:34 smithi017 ceph-mon[29506]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-02-14T17:17:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:34 smithi017 ceph-mon[29506]: osdmap e26: 4 total, 4 up, 4 in 2024-02-14T17:17:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:35 smithi111 ceph-mon[25603]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:35 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:35 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:35 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:36.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[24769]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[29506]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:35 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:36 smithi111 ceph-mon[25603]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:37.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:36 smithi017 ceph-mon[24769]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:37.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:36 smithi017 ceph-mon[29506]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/575895820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]: dispatch 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]: dispatch 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]': finished 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[24769]: osdmap e27: 5 total, 4 up, 5 in 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/1147070342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/575895820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]: dispatch 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]: dispatch 2024-02-14T17:17:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]': finished 2024-02-14T17:17:38.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[29506]: osdmap e27: 5 total, 4 up, 5 in 2024-02-14T17:17:38.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:38.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:37 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/1147070342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:17:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:37 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/575895820' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]: dispatch 2024-02-14T17:17:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:37 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]: dispatch 2024-02-14T17:17:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:37 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82f83614-981c-4b5f-bdba-00241f18bbb9"}]': finished 2024-02-14T17:17:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:37 smithi111 ceph-mon[25603]: osdmap e27: 5 total, 4 up, 5 in 2024-02-14T17:17:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:37 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:37 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/1147070342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:17:39.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:39 smithi017 ceph-mon[24769]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:39 smithi017 ceph-mon[29506]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:39 smithi111 ceph-mon[25603]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:41.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:41 smithi017 ceph-mon[24769]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:41.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:41 smithi017 ceph-mon[29506]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:41 smithi111 ceph-mon[25603]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:43 smithi017 ceph-mon[24769]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:43 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:17:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:43 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:43.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:43 smithi017 ceph-mon[29506]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:43.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:43 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:17:43.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:43 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:43 smithi111 ceph-mon[25603]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:43 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:17:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:43 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:44.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:44 smithi017 ceph-mon[24769]: Deploying daemon osd.4 on smithi111 2024-02-14T17:17:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:44 smithi017 ceph-mon[29506]: Deploying daemon osd.4 on smithi111 2024-02-14T17:17:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:44 smithi111 ceph-mon[25603]: Deploying daemon osd.4 on smithi111 2024-02-14T17:17:45.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:45 smithi017 ceph-mon[24769]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:45.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:45 smithi017 ceph-mon[29506]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:45.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:45 smithi111 ceph-mon[25603]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:47.553 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:47 smithi111 ceph-mon[25603]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:47.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:47 smithi017 ceph-mon[24769]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:47.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:47 smithi017 ceph-mon[29506]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:49.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:49 smithi111 ceph-mon[25603]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:49.396 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:49.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:49.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:49.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:49 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[29506]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[24769]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:49.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:49.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:49 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:50.776 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 4 on host 'smithi111' 2024-02-14T17:17:51.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:51 smithi111 ceph-mon[25603]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:51.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:51 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:51.354 DEBUG:teuthology.orchestra.run.smithi111:osd.4> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.4.service 2024-02-14T17:17:51.356 INFO:tasks.cephadm:Deploying osd.5 on smithi111 with /dev/vg_nvme/lv_3... 2024-02-14T17:17:51.356 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-14T17:17:51.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:51 smithi017 ceph-mon[24769]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:51 smithi017 ceph-mon[29506]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:52.105 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:52 smithi111 ceph-mon[25603]: from='osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:17:52.106 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:52 smithi111 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:17:52.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:52 smithi017 ceph-mon[24769]: from='osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:17:52.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:52 smithi017 ceph-mon[24769]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:17:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:52 smithi017 ceph-mon[29506]: from='osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:17:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:52 smithi017 ceph-mon[29506]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:17:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-mon[25603]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-14T17:17:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-mon[25603]: osdmap e28: 5 total, 4 up, 5 in 2024-02-14T17:17:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-mon[25603]: from='osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:17:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:17:53.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:53.235 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[29025]: 2024-02-14T17:17:53.117+0000 7f9d0c008700 -1 osd.4 0 waiting for initial osdmap 2024-02-14T17:17:53.235 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:17:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[29025]: 2024-02-14T17:17:53.121+0000 7f9d0699f700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:17:53.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[24769]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[24769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[24769]: osdmap e28: 5 total, 4 up, 5 in 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[24769]: from='osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[24769]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[29506]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[29506]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[29506]: osdmap e28: 5 total, 4 up, 5 in 2024-02-14T17:17:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:53.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[29506]: from='osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:17:53.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[29506]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:17:53.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:53 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:53.645 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-14T17:17:53.645 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-14T17:17:53.645 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-02-14T17:17:53.645 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-02-14T17:17:53.645 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122137 s, 859 MB/s 2024-02-14T17:17:53.645 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-02-14T17:17:54.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:54 smithi111 ceph-mon[25603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:17:54.331 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:54 smithi111 ceph-mon[25603]: osdmap e29: 5 total, 4 up, 5 in 2024-02-14T17:17:54.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:54 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:54.332 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:54 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:54.333 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-02-14T17:17:54.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[24769]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:17:54.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[24769]: osdmap e29: 5 total, 4 up, 5 in 2024-02-14T17:17:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[29506]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:17:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[29506]: osdmap e29: 5 total, 4 up, 5 in 2024-02-14T17:17:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:54 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:55.226 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:55 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:17:55.227 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:55 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:17:55.227 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:55 smithi111 ceph-mon[25603]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:55.227 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:55 smithi111 ceph-mon[25603]: osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102] boot 2024-02-14T17:17:55.227 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:55 smithi111 ceph-mon[25603]: osdmap e30: 5 total, 5 up, 5 in 2024-02-14T17:17:55.227 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:55 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:55.227 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:55 smithi111 ceph-mon[25603]: osdmap e31: 5 total, 5 up, 5 in 2024-02-14T17:17:55.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[24769]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[24769]: osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102] boot 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[24769]: osdmap e30: 5 total, 5 up, 5 in 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[24769]: osdmap e31: 5 total, 5 up, 5 in 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[29506]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-02-14T17:17:55.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[29506]: osd.4 [v2:172.21.15.111:6800/2508759102,v1:172.21.15.111:6801/2508759102] boot 2024-02-14T17:17:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[29506]: osdmap e30: 5 total, 5 up, 5 in 2024-02-14T17:17:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:17:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:55 smithi017 ceph-mon[29506]: osdmap e31: 5 total, 5 up, 5 in 2024-02-14T17:17:57.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:56 smithi111 ceph-mon[25603]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:17:57.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:56 smithi111 ceph-mon[25603]: osdmap e32: 5 total, 5 up, 5 in 2024-02-14T17:17:57.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:56 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:57.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:56 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:57.019 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:56 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:57.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[24769]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[24769]: osdmap e32: 5 total, 5 up, 5 in 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[29506]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[29506]: osdmap e32: 5 total, 5 up, 5 in 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:17:57.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:56 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: osdmap e33: 5 total, 5 up, 5 in 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi111 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi111 to 17088M 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:58.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:58.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:58.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:57 smithi111 ceph-mon[25603]: osdmap e34: 5 total, 5 up, 5 in 2024-02-14T17:17:58.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: osdmap e33: 5 total, 5 up, 5 in 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi111 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi111 to 17088M 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[24769]: osdmap e34: 5 total, 5 up, 5 in 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: osdmap e33: 5 total, 5 up, 5 in 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi111 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi111 to 17088M 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:17:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:57 smithi017 ceph-mon[29506]: osdmap e34: 5 total, 5 up, 5 in 2024-02-14T17:17:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:59 smithi017 ceph-mon[29506]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-14T17:17:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:59 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/3931396222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]: dispatch 2024-02-14T17:17:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:17:59 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]: dispatch 2024-02-14T17:17:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:59 smithi017 ceph-mon[24769]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-14T17:17:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:59 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/3931396222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]: dispatch 2024-02-14T17:17:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:17:59 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]: dispatch 2024-02-14T17:17:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:59 smithi111 ceph-mon[25603]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-14T17:17:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:59 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/3931396222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]: dispatch 2024-02-14T17:17:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:17:59 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]: dispatch 2024-02-14T17:18:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]': finished 2024-02-14T17:18:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[24769]: osdmap e35: 6 total, 5 up, 6 in 2024-02-14T17:18:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/1901815674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:00.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]': finished 2024-02-14T17:18:00.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[29506]: osdmap e35: 6 total, 5 up, 6 in 2024-02-14T17:18:00.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:00 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/1901815674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:00 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c46f35f-fb44-46f9-a078-20ff42d28ce8"}]': finished 2024-02-14T17:18:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:00 smithi111 ceph-mon[25603]: osdmap e35: 6 total, 5 up, 6 in 2024-02-14T17:18:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:00 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/1901815674' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:01.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:01 smithi017 ceph-mon[29506]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-14T17:18:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:01 smithi017 ceph-mon[24769]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-14T17:18:01.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:01 smithi111 ceph-mon[25603]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-02-14T17:18:03.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:03 smithi111 ceph-mon[25603]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-14T17:18:03.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:03 smithi017 ceph-mon[24769]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-14T17:18:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:03 smithi017 ceph-mon[29506]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-02-14T17:18:05.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:05 smithi111 ceph-mon[25603]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-14T17:18:05.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:05 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:18:05.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:05 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:05.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:05 smithi017 ceph-mon[24769]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-14T17:18:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:05 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:18:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:05 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:05 smithi017 ceph-mon[29506]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-02-14T17:18:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:05 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:18:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:05 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:06.661 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:06 smithi111 ceph-mon[25603]: Deploying daemon osd.5 on smithi111 2024-02-14T17:18:06.661 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:18:06.661 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:18:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:06 smithi017 ceph-mon[24769]: Deploying daemon osd.5 on smithi111 2024-02-14T17:18:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:18:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:18:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:06 smithi017 ceph-mon[29506]: Deploying daemon osd.5 on smithi111 2024-02-14T17:18:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:18:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:18:07.674 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:07 smithi111 ceph-mon[25603]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:07.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:07 smithi017 ceph-mon[24769]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:07.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:07 smithi017 ceph-mon[29506]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:08.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:08 smithi111 ceph-mon[25603]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:08.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:08 smithi017 ceph-mon[24769]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:08 smithi017 ceph-mon[29506]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:11.357 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:11 smithi111 ceph-mon[25603]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:11.358 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:11.358 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:11.358 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:11.358 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:11 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:11.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[24769]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[29506]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:11.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:11 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:13.123 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:13 smithi111 ceph-mon[25603]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:13.157 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 5 on host 'smithi111' 2024-02-14T17:18:13.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:13 smithi017 ceph-mon[24769]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:13.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:13 smithi017 ceph-mon[29506]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:13.729 DEBUG:teuthology.orchestra.run.smithi111:osd.5> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.5.service 2024-02-14T17:18:13.732 INFO:tasks.cephadm:Deploying osd.6 on smithi111 with /dev/vg_nvme/lv_2... 2024-02-14T17:18:13.732 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-14T17:18:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:14 smithi111 ceph-mon[25603]: from='osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:18:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:14 smithi111 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:18:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:14.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[24769]: from='osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:18:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[24769]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:18:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[29506]: from='osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:18:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[29506]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:18:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:15.355 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-mon[25603]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:15.356 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-14T17:18:15.356 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-mon[25603]: osdmap e36: 6 total, 5 up, 6 in 2024-02-14T17:18:15.356 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:15.356 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-mon[25603]: from='osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:15.356 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:15.356 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[31912]: 2024-02-14T17:18:15.050+0000 7f4917be4700 -1 osd.5 0 waiting for initial osdmap 2024-02-14T17:18:15.356 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:18:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[31912]: 2024-02-14T17:18:15.055+0000 7f491257b700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:18:15.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[24769]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[24769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[24769]: osdmap e36: 6 total, 5 up, 6 in 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[24769]: from='osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[24769]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[29506]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[29506]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[29506]: osdmap e36: 6 total, 5 up, 6 in 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[29506]: from='osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:15 smithi017 ceph-mon[29506]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:16.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:16 smithi111 ceph-mon[25603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:18:16.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:16 smithi111 ceph-mon[25603]: osdmap e37: 6 total, 5 up, 6 in 2024-02-14T17:18:16.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:16 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:16.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:16 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-14T17:18:16.262 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-14T17:18:16.262 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-02-14T17:18:16.263 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-02-14T17:18:16.263 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122605 s, 855 MB/s 2024-02-14T17:18:16.263 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[24769]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[24769]: osdmap e37: 6 total, 5 up, 6 in 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[29506]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[29506]: osdmap e37: 6 total, 5 up, 6 in 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:16 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[24769]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[24769]: osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094] boot 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[24769]: osdmap e38: 6 total, 6 up, 6 in 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:17.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:18:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:18:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[29506]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[29506]: osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094] boot 2024-02-14T17:18:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[29506]: osdmap e38: 6 total, 6 up, 6 in 2024-02-14T17:18:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:17 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:17.443 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:17 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:18:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:17 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:18:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:17 smithi111 ceph-mon[25603]: pgmap v122: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-02-14T17:18:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:17 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:17 smithi111 ceph-mon[25603]: osd.5 [v2:172.21.15.111:6808/441844094,v1:172.21.15.111:6809/441844094] boot 2024-02-14T17:18:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:17 smithi111 ceph-mon[25603]: osdmap e38: 6 total, 6 up, 6 in 2024-02-14T17:18:17.444 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:17 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:18:17.469 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-02-14T17:18:18.273 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:18 smithi111 ceph-mon[25603]: osdmap e39: 6 total, 6 up, 6 in 2024-02-14T17:18:18.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:18 smithi017 ceph-mon[24769]: osdmap e39: 6 total, 6 up, 6 in 2024-02-14T17:18:18.484 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:18 smithi017 ceph-mon[29506]: osdmap e39: 6 total, 6 up, 6 in 2024-02-14T17:18:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-14T17:18:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: osdmap e40: 6 total, 6 up, 6 in 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi111 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi111 to 8544M 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:19.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:19 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: osdmap e40: 6 total, 6 up, 6 in 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi111 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi111 to 8544M 2024-02-14T17:18:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: pgmap v125: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: osdmap e40: 6 total, 6 up, 6 in 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi111 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:19.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi111 to 8544M 2024-02-14T17:18:19.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:19.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:19.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:19.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:19.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:19 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:20.228 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:20 smithi111 ceph-mon[25603]: osdmap e41: 6 total, 6 up, 6 in 2024-02-14T17:18:20.228 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:20 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:18:20.228 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:20 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:18:20.228 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:20 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[24769]: osdmap e41: 6 total, 6 up, 6 in 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[29506]: osdmap e41: 6 total, 6 up, 6 in 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:18:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:20 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:21.120 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:21 smithi111 ceph-mon[25603]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:18:21.121 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:21 smithi111 ceph-mon[25603]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:21.121 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:21 smithi111 ceph-mon[25603]: osdmap e42: 6 total, 6 up, 6 in 2024-02-14T17:18:21.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:21 smithi017 ceph-mon[24769]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:18:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:21 smithi017 ceph-mon[24769]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:21 smithi017 ceph-mon[24769]: osdmap e42: 6 total, 6 up, 6 in 2024-02-14T17:18:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:21 smithi017 ceph-mon[29506]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:18:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:21 smithi017 ceph-mon[29506]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:21 smithi017 ceph-mon[29506]: osdmap e42: 6 total, 6 up, 6 in 2024-02-14T17:18:22.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/1817679042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]: dispatch 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]: dispatch 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]': finished 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[24769]: osdmap e43: 7 total, 6 up, 7 in 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/1817679042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]: dispatch 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]: dispatch 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]': finished 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[29506]: osdmap e43: 7 total, 6 up, 7 in 2024-02-14T17:18:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:22 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:22 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/1817679042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]: dispatch 2024-02-14T17:18:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:22 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]: dispatch 2024-02-14T17:18:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:22 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c5091bd-4021-4c53-91d3-5a9bdebb85b0"}]': finished 2024-02-14T17:18:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:22 smithi111 ceph-mon[25603]: osdmap e43: 7 total, 6 up, 7 in 2024-02-14T17:18:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:22 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:23.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:23 smithi017 ceph-mon[24769]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:23.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:23 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/94383042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:23.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:23 smithi017 ceph-mon[29506]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:23.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:23 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/94383042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:23 smithi111 ceph-mon[25603]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:23 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/94383042' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:24.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:24 smithi017 ceph-mon[24769]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-14T17:18:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:24 smithi017 ceph-mon[24769]: Cluster is now healthy 2024-02-14T17:18:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:24 smithi017 ceph-mon[29506]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-14T17:18:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:24 smithi017 ceph-mon[29506]: Cluster is now healthy 2024-02-14T17:18:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:24 smithi111 ceph-mon[25603]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-14T17:18:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:24 smithi111 ceph-mon[25603]: Cluster is now healthy 2024-02-14T17:18:25.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:25 smithi017 ceph-mon[24769]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-14T17:18:25.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:25 smithi017 ceph-mon[29506]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-14T17:18:25.461 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:25 smithi111 ceph-mon[25603]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-02-14T17:18:27.328 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:27 smithi111 ceph-mon[25603]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-14T17:18:27.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:27 smithi017 ceph-mon[24769]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-14T17:18:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:27 smithi017 ceph-mon[29506]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-02-14T17:18:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:28 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:18:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:28 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:28.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:18:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:28 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:28 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:18:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:28 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:29.433 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:29 smithi111 ceph-mon[25603]: Deploying daemon osd.6 on smithi111 2024-02-14T17:18:29.433 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:29 smithi111 ceph-mon[25603]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:18:29.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:29 smithi017 ceph-mon[24769]: Deploying daemon osd.6 on smithi111 2024-02-14T17:18:29.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:29 smithi017 ceph-mon[24769]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:18:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:29 smithi017 ceph-mon[29506]: Deploying daemon osd.6 on smithi111 2024-02-14T17:18:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:29 smithi017 ceph-mon[29506]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:18:31.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:31 smithi017 ceph-mon[24769]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:31.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:31 smithi017 ceph-mon[29506]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:31.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:31 smithi111 ceph-mon[25603]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:33.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[24769]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[29506]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:33 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:33.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:33 smithi111 ceph-mon[25603]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:33.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:33.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:33.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:33.481 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:33 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:35.379 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:35 smithi111 ceph-mon[25603]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:35.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:35 smithi017 ceph-mon[24769]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:35 smithi017 ceph-mon[29506]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:36.191 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 6 on host 'smithi111' 2024-02-14T17:18:36.472 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.472 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.472 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:18:36.472 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:18:36.473 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.473 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:36.473 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:36.473 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:36 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:18:36.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:18:36.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:36.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:36.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:36.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:36 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:36.715 DEBUG:teuthology.orchestra.run.smithi111:osd.6> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.6.service 2024-02-14T17:18:36.717 INFO:tasks.cephadm:Deploying osd.7 on smithi111 with /dev/vg_nvme/lv_1... 2024-02-14T17:18:36.717 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-14T17:18:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-mon[25603]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-14T17:18:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-mon[25603]: osdmap e44: 7 total, 6 up, 7 in 2024-02-14T17:18:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-mon[25603]: from='osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[24769]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[24769]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[24769]: osdmap e44: 7 total, 6 up, 7 in 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[24769]: from='osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[24769]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[29506]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[29506]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-14T17:18:37.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[29506]: osdmap e44: 7 total, 6 up, 7 in 2024-02-14T17:18:37.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:37.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[29506]: from='osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:37.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:37 smithi017 ceph-mon[29506]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:18:38.142 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[34822]: 2024-02-14T17:18:37.863+0000 7f8772319700 -1 osd.6 0 waiting for initial osdmap 2024-02-14T17:18:38.142 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:18:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[34822]: 2024-02-14T17:18:37.872+0000 7f876acac700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:18:39.073 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:38 smithi111 ceph-mon[25603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:18:39.074 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:38 smithi111 ceph-mon[25603]: osdmap e45: 7 total, 6 up, 7 in 2024-02-14T17:18:39.074 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:38 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:39.074 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:38 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:39.074 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:38 smithi111 ceph-mon[25603]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[29506]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:18:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[29506]: osdmap e45: 7 total, 6 up, 7 in 2024-02-14T17:18:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[29506]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:39.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[24769]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:18:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[24769]: osdmap e45: 7 total, 6 up, 7 in 2024-02-14T17:18:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:38 smithi017 ceph-mon[24769]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-02-14T17:18:39.595 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-14T17:18:39.595 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-14T17:18:39.595 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-02-14T17:18:39.595 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-02-14T17:18:39.595 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120167 s, 873 MB/s 2024-02-14T17:18:39.595 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-02-14T17:18:39.894 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:39 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:18:39.894 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:39 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:18:39.894 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:39 smithi111 ceph-mon[25603]: osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343] boot 2024-02-14T17:18:39.894 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:39 smithi111 ceph-mon[25603]: osdmap e46: 7 total, 7 up, 7 in 2024-02-14T17:18:39.894 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:39 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:40.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[24769]: osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343] boot 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[24769]: osdmap e46: 7 total, 7 up, 7 in 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[29506]: osd.6 [v2:172.21.15.111:6816/3964813343,v1:172.21.15.111:6817/3964813343] boot 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[29506]: osdmap e46: 7 total, 7 up, 7 in 2024-02-14T17:18:40.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:39 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:18:40.278 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2024-02-14T17:18:41.136 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:40 smithi111 ceph-mon[25603]: osdmap e47: 7 total, 7 up, 7 in 2024-02-14T17:18:41.136 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:40 smithi111 ceph-mon[25603]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:41.136 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:40 smithi111 ceph-mon[25603]: osdmap e48: 7 total, 7 up, 7 in 2024-02-14T17:18:41.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:40 smithi017 ceph-mon[24769]: osdmap e47: 7 total, 7 up, 7 in 2024-02-14T17:18:41.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:40 smithi017 ceph-mon[24769]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:41.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:40 smithi017 ceph-mon[24769]: osdmap e48: 7 total, 7 up, 7 in 2024-02-14T17:18:41.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:40 smithi017 ceph-mon[29506]: osdmap e47: 7 total, 7 up, 7 in 2024-02-14T17:18:41.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:40 smithi017 ceph-mon[29506]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:41.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:40 smithi017 ceph-mon[29506]: osdmap e48: 7 total, 7 up, 7 in 2024-02-14T17:18:42.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi111 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi111 to 5696M 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[29506]: osdmap e49: 7 total, 7 up, 7 in 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi111 2024-02-14T17:18:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi111 to 5696M 2024-02-14T17:18:42.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:42 smithi017 ceph-mon[24769]: osdmap e49: 7 total, 7 up, 7 in 2024-02-14T17:18:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi111 2024-02-14T17:18:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:18:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi111 to 5696M 2024-02-14T17:18:42.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:42.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:42 smithi111 ceph-mon[25603]: osdmap e49: 7 total, 7 up, 7 in 2024-02-14T17:18:43.272 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:43 smithi111 ceph-mon[25603]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:43.273 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:43 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:18:43.273 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:43 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:18:43.273 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:43 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:43.273 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:43 smithi111 ceph-mon[25603]: osdmap e50: 7 total, 7 up, 7 in 2024-02-14T17:18:43.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[24769]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[24769]: osdmap e50: 7 total, 7 up, 7 in 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[29506]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:43.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:43 smithi017 ceph-mon[29506]: osdmap e50: 7 total, 7 up, 7 in 2024-02-14T17:18:44.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:44 smithi017 ceph-mon[24769]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:18:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:44 smithi017 ceph-mon[29506]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:18:44.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:44 smithi111 ceph-mon[25603]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:18:45.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[24769]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:45.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/3903248993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]: dispatch 2024-02-14T17:18:45.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]: dispatch 2024-02-14T17:18:45.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[24769]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]': finished 2024-02-14T17:18:45.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[24769]: osdmap e51: 8 total, 7 up, 8 in 2024-02-14T17:18:45.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:18:45.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[29506]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:45.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/3903248993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]: dispatch 2024-02-14T17:18:45.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]: dispatch 2024-02-14T17:18:45.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[29506]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]': finished 2024-02-14T17:18:45.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[29506]: osdmap e51: 8 total, 7 up, 8 in 2024-02-14T17:18:45.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:45 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:18:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:45 smithi111 ceph-mon[25603]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:45 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/3903248993' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]: dispatch 2024-02-14T17:18:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:45 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]: dispatch 2024-02-14T17:18:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:45 smithi111 ceph-mon[25603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9cc83c8-5fd6-4e76-92df-c7c15b15b86d"}]': finished 2024-02-14T17:18:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:45 smithi111 ceph-mon[25603]: osdmap e51: 8 total, 7 up, 8 in 2024-02-14T17:18:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:45 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:18:46.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:46 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/714128993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:46 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/714128993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:46 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/714128993' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-14T17:18:47.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:47 smithi017 ceph-mon[24769]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-14T17:18:47.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:47 smithi017 ceph-mon[29506]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-14T17:18:47.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:47 smithi111 ceph-mon[25603]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-02-14T17:18:49.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:49 smithi111 ceph-mon[25603]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-14T17:18:49.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:49 smithi017 ceph-mon[24769]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-14T17:18:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:49 smithi017 ceph-mon[29506]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-14T17:18:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:51 smithi111 ceph-mon[25603]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:18:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:51 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:18:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:51 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:51.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:51 smithi017 ceph-mon[24769]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:18:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:18:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:51 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:51 smithi017 ceph-mon[29506]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:18:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:18:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:51 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:52.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:52 smithi017 ceph-mon[24769]: Deploying daemon osd.7 on smithi111 2024-02-14T17:18:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:52 smithi017 ceph-mon[29506]: Deploying daemon osd.7 on smithi111 2024-02-14T17:18:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:52 smithi111 ceph-mon[25603]: Deploying daemon osd.7 on smithi111 2024-02-14T17:18:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:53 smithi017 ceph-mon[29506]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-02-14T17:18:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:53 smithi017 ceph-mon[24769]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-02-14T17:18:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:53 smithi111 ceph-mon[25603]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-02-14T17:18:55.362 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:55 smithi111 ceph-mon[25603]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:55.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:55 smithi017 ceph-mon[24769]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:55.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:55 smithi017 ceph-mon[29506]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-02-14T17:18:57.184 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:56 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:57.184 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:56 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:57.185 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:56 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:57.185 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:56 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:57.185 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:56 smithi111 ceph-mon[25603]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:57.189 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:57.189 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[29506]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:18:57.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:56 smithi017 ceph-mon[24769]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:58.888 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 7 on host 'smithi111' 2024-02-14T17:18:59.210 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:59 smithi111 ceph-mon[25603]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:59.210 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:59 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:59.211 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:59 smithi111 ceph-mon[25603]: from='osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:18:59.211 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:18:59 smithi111 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:18:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[29506]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[29506]: from='osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:18:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[29506]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:18:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[24769]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:18:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:18:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[24769]: from='osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:18:59.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:18:59 smithi017 ceph-mon[24769]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:18:59.518 DEBUG:teuthology.orchestra.run.smithi111:osd.7> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.7.service 2024-02-14T17:18:59.521 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-14T17:18:59.522 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-14T17:19:00.976 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:00.976 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-14T17:19:00.977 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-mon[25603]: osdmap e52: 8 total, 7 up, 8 in 2024-02-14T17:19:00.977 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:00.977 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-mon[25603]: from='osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:19:00.977 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:19:00.977 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-mon[25603]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:19:00.977 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[37812]: 2024-02-14T17:19:00.893+0000 7ff374f6c700 -1 osd.7 0 waiting for initial osdmap 2024-02-14T17:19:00.977 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:19:00 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[37812]: 2024-02-14T17:19:00.902+0000 7ff36e901700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:19:01.036 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:01.036 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[29506]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-14T17:19:01.036 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[29506]: osdmap e52: 8 total, 7 up, 8 in 2024-02-14T17:19:01.036 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:01.036 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[29506]: from='osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:19:01.036 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[29506]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:19:01.036 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[29506]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:19:01.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:01.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[24769]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-14T17:19:01.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[24769]: osdmap e52: 8 total, 7 up, 8 in 2024-02-14T17:19:01.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:01.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[24769]: from='osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:19:01.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[24769]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:19:01.037 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:00 smithi017 ceph-mon[24769]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-02-14T17:19:01.643 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:19:01.893 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[24769]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:19:02.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[24769]: osdmap e53: 8 total, 7 up, 8 in 2024-02-14T17:19:02.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:02.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:02.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3643390042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:19:02.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[29506]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:19:02.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[29506]: osdmap e53: 8 total, 7 up, 8 in 2024-02-14T17:19:02.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:02.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:02.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:01 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3643390042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:19:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:01 smithi111 ceph-mon[25603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-02-14T17:19:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:01 smithi111 ceph-mon[25603]: osdmap e53: 8 total, 7 up, 8 in 2024-02-14T17:19:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:01 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:01 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:01 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3643390042' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:19:02.281 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1707931118,"num_in_osds":8,"osd_in_since":1707931124,"num_remapped_pgs":0} 2024-02-14T17:19:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: purged_snaps scrub starts 2024-02-14T17:19:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: purged_snaps scrub ok 2024-02-14T17:19:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482] boot 2024-02-14T17:19:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: osdmap e54: 8 total, 8 up, 8 in 2024-02-14T17:19:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:02.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:02 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: purged_snaps scrub starts 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: purged_snaps scrub ok 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482] boot 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: osdmap e54: 8 total, 8 up, 8 in 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: purged_snaps scrub starts 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: purged_snaps scrub ok 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: osd.7 [v2:172.21.15.111:6824/661554482,v1:172.21.15.111:6825/661554482] boot 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: osdmap e54: 8 total, 8 up, 8 in 2024-02-14T17:19:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:03.193 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:02 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:03.282 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd stat -f json 2024-02-14T17:19:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:03 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi111 2024-02-14T17:19:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:03 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi111 to 4272M 2024-02-14T17:19:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:03 smithi017 ceph-mon[24769]: osdmap e55: 8 total, 8 up, 8 in 2024-02-14T17:19:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:03 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi111 2024-02-14T17:19:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:03 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi111 to 4272M 2024-02-14T17:19:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:03 smithi017 ceph-mon[29506]: osdmap e55: 8 total, 8 up, 8 in 2024-02-14T17:19:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:03 smithi111 ceph-mon[25603]: Detected new or changed devices on smithi111 2024-02-14T17:19:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:03 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi111 to 4272M 2024-02-14T17:19:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:03 smithi111 ceph-mon[25603]: osdmap e55: 8 total, 8 up, 8 in 2024-02-14T17:19:04.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:04 smithi017 ceph-mon[24769]: osdmap e56: 8 total, 8 up, 8 in 2024-02-14T17:19:04.971 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:04 smithi017 ceph-mon[24769]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:04.971 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:04 smithi017 ceph-mon[29506]: osdmap e56: 8 total, 8 up, 8 in 2024-02-14T17:19:04.972 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:04 smithi017 ceph-mon[29506]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:04 smithi111 ceph-mon[25603]: osdmap e56: 8 total, 8 up, 8 in 2024-02-14T17:19:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:04 smithi111 ceph-mon[25603]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:05.368 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:19:05.987 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1707931141,"num_in_osds":8,"osd_in_since":1707931124,"num_remapped_pgs":0} 2024-02-14T17:19:05.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-14T17:19:06.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:05 smithi017 ceph-mon[24769]: osdmap e57: 8 total, 8 up, 8 in 2024-02-14T17:19:06.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:05 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/99623189' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:19:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:05 smithi017 ceph-mon[29506]: osdmap e57: 8 total, 8 up, 8 in 2024-02-14T17:19:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:05 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/99623189' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:19:06.206 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:05 smithi111 ceph-mon[25603]: osdmap e57: 8 total, 8 up, 8 in 2024-02-14T17:19:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:05 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/99623189' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-14T17:19:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[24769]: osdmap e58: 8 total, 8 up, 8 in 2024-02-14T17:19:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[24769]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:07.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[29506]: osdmap e58: 8 total, 8 up, 8 in 2024-02-14T17:19:07.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[29506]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:07.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:07.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:06 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:06 smithi111 ceph-mon[25603]: osdmap e58: 8 total, 8 up, 8 in 2024-02-14T17:19:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:06 smithi111 ceph-mon[25603]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:06 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:07.986 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:19:07.986 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":58,"fsid":"612435b4-cb5c-11ee-95b9-87774f69a715","created":"2024-02-14T17:14:10.781716+0000","modified":"2024-02-14T17:19:05.905770+0000","last_up_change":"2024-02-14T17:19:01.887456+0000","last_in_change":"2024-02-14T17:18:44.869711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-14T17:17:10.259165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a980d777-c667-4148-b5fa-b9fc89d6dc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6803","nonce":2126863272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6805","nonce":2126863272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6809","nonce":2126863272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6807","nonce":2126863272}]},"public_addr":"172.21.15.17:6803/2126863272","cluster_addr":"172.21.15.17:6805/2126863272","heartbeat_back_addr":"172.21.15.17:6809/2126863272","heartbeat_front_addr":"172.21.15.17:6807/2126863272","state":["exists","up"]},{"osd":1,"uuid":"517877bd-896e-4e1d-93e6-2a704bca2e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6811","nonce":336965104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6813","nonce":336965104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6817","nonce":336965104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6815","nonce":336965104}]},"public_addr":"172.21.15.17:6811/336965104","cluster_addr":"172.21.15.17:6813/336965104","heartbeat_back_addr":"172.21.15.17:6817/336965104","heartbeat_front_addr":"172.21.15.17:6815/336965104","state":["exists","up"]},{"osd":2,"uuid":"c810adb5-4205-4e6a-a102-8cdeafeb7964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6819","nonce":3232569458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6821","nonce":3232569458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6825","nonce":3232569458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6823","nonce":3232569458}]},"public_addr":"172.21.15.17:6819/3232569458","cluster_addr":"172.21.15.17:6821/3232569458","heartbeat_back_addr":"172.21.15.17:6825/3232569458","heartbeat_front_addr":"172.21.15.17:6823/3232569458","state":["exists","up"]},{"osd":3,"uuid":"0c10a4f3-4cd7-44da-8859-96c78d7b5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6827","nonce":2789642823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6829","nonce":2789642823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6833","nonce":2789642823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6831","nonce":2789642823}]},"public_addr":"172.21.15.17:6827/2789642823","cluster_addr":"172.21.15.17:6829/2789642823","heartbeat_back_addr":"172.21.15.17:6833/2789642823","heartbeat_front_addr":"172.21.15.17:6831/2789642823","state":["exists","up"]},{"osd":4,"uuid":"82f83614-981c-4b5f-bdba-00241f18bbb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6801","nonce":2508759102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6803","nonce":2508759102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6807","nonce":2508759102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6805","nonce":2508759102}]},"public_addr":"172.21.15.111:6801/2508759102","cluster_addr":"172.21.15.111:6803/2508759102","heartbeat_back_addr":"172.21.15.111:6807/2508759102","heartbeat_front_addr":"172.21.15.111:6805/2508759102","state":["exists","up"]},{"osd":5,"uuid":"6c46f35f-fb44-46f9-a078-20ff42d28ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6809","nonce":441844094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6811","nonce":441844094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6815","nonce":441844094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6813","nonce":441844094}]},"public_addr":"172.21.15.111:6809/441844094","cluster_addr":"172.21.15.111:6811/441844094","heartbeat_back_addr":"172.21.15.111:6815/441844094","heartbeat_front_addr":"172.21.15.111:6813/441844094","state":["exists","up"]},{"osd":6,"uuid":"3c5091bd-4021-4c53-91d3-5a9bdebb85b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6817","nonce":3964813343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6819","nonce":3964813343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6823","nonce":3964813343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6821","nonce":3964813343}]},"public_addr":"172.21.15.111:6817/3964813343","cluster_addr":"172.21.15.111:6819/3964813343","heartbeat_back_addr":"172.21.15.111:6823/3964813343","heartbeat_front_addr":"172.21.15.111:6821/3964813343","state":["exists","up"]},{"osd":7,"uuid":"a9cc83c8-5fd6-4e76-92df-c7c15b15b86d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6825","nonce":661554482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6827","nonce":661554482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6831","nonce":661554482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6829","nonce":661554482}]},"public_addr":"172.21.15.111:6825/661554482","cluster_addr":"172.21.15.111:6827/661554482","heartbeat_back_addr":"172.21.15.111:6831/661554482","heartbeat_front_addr":"172.21.15.111:6829/661554482","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:16:22.794805+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:16:45.311188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:07.896583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:30.997825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:52.265643+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:18:14.142700+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:18:36.986785+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:19:00.036720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2689455788":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/1905974843":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/996094750":"2024-02-15T17:14:42.920120+0000","172.21.15.17:0/1219630449":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/1994288429":"2024-02-15T17:14:42.920120+0000","172.21.15.17:6800/1397676400":"2024-02-15T17:14:42.920120+0000","172.21.15.17:6801/1666345519":"2024-02-15T17:15:05.331154+0000","172.21.15.17:6801/1397676400":"2024-02-15T17:14:42.920120+0000","172.21.15.17:6800/1666345519":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/808657630":"2024-02-15T17:14:42.920120+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-14T17:19:08.267 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:08 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2496964017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:08.267 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:08 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2496964017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:08 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2496964017' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:08.626 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-14T17:17:10.259165+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-14T17:19:08.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-14T17:19:08.846 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:09.254 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:09 smithi017 ceph-mon[24769]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-02-14T17:19:09.254 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:09 smithi017 ceph-mon[29506]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-02-14T17:19:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:09 smithi111 ceph-mon[25603]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-02-14T17:19:10.661 INFO:teuthology.orchestra.run.smithi017.stdout:pg_num: 1 2024-02-14T17:19:11.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:11 smithi017 ceph-mon[29506]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-14T17:19:11.308 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:11 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1773524462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-14T17:19:11.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:11 smithi017 ceph-mon[24769]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-14T17:19:11.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:11 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1773524462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-14T17:19:11.334 INFO:tasks.cephadm:Adding prometheus.a on smithi111 2024-02-14T17:19:11.334 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch apply prometheus '1;smithi111=a' 2024-02-14T17:19:11.365 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:11 smithi111 ceph-mon[25603]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-02-14T17:19:11.365 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:11 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1773524462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-14T17:19:13.050 INFO:teuthology.orchestra.run.smithi111.stdout:Scheduled prometheus update... 2024-02-14T17:19:13.309 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:13 smithi111 ceph-mon[25603]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:19:13.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:13 smithi017 ceph-mon[24769]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:19:13.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:13 smithi017 ceph-mon[29506]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-02-14T17:19:13.663 DEBUG:teuthology.orchestra.run.smithi111:prometheus.a> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service 2024-02-14T17:19:13.705 INFO:tasks.cephadm:Adding node-exporter.a on smithi017 2024-02-14T17:19:13.706 INFO:tasks.cephadm:Adding node-exporter.b on smithi111 2024-02-14T17:19:13.706 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch apply node-exporter '2;smithi017=a;smithi111=b' 2024-02-14T17:19:14.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: Saving service prometheus spec with placement smithi111=a;count:1 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ignoring --setuser ceph since I am not root 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ignoring --setgroup ceph since I am not root 2024-02-14T17:19:14.327 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-mgr[26785]: -- 172.21.15.111:0/3732772588 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bd5a8e0340 con 0x55bd5a8b8800 2024-02-14T17:19:14.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: Saving service prometheus spec with placement smithi111=a;count:1 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-14T17:19:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: Saving service prometheus spec with placement smithi111=a;count:1 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setuser ceph since I am not root 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setgroup ceph since I am not root 2024-02-14T17:19:14.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-mgr[25006]: -- 172.21.15.17:0/137066725 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562bacc5e340 con 0x562bacc36c00 2024-02-14T17:19:14.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:14.608+0000 7f8f0f123000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:19:14.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:14.699+0000 7f8f0f123000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:19:14.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:14.607+0000 7f64d61d4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:19:14.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:14 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:14.698+0000 7f64d61d4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:19:15.284 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-mon[25603]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-14T17:19:15.284 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-mon[25603]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-14T17:19:15.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-mon[24769]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-14T17:19:15.394 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-mon[24769]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-14T17:19:15.394 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-mon[29506]: from='mgr.14150 172.21.15.17:0/3411399825' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-14T17:19:15.394 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-mon[29506]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-14T17:19:15.394 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:15.302+0000 7f8f0f123000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:19:15.394 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:15.392+0000 7f8f0f123000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:19:15.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:15.299+0000 7f64d61d4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:19:15.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:15.390+0000 7f64d61d4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:19:15.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:15.477+0000 7f64d61d4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:19:15.635 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:15.632+0000 7f64d61d4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:19:15.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:15.476+0000 7f8f0f123000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:19:15.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:15.627+0000 7f8f0f123000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:19:15.908 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:15.716+0000 7f64d61d4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:19:15.908 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:15.823+0000 7f64d61d4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:19:15.909 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:15 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:15.906+0000 7f64d61d4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:19:16.071 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:15.709+0000 7f8f0f123000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:19:16.071 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:15.820+0000 7f8f0f123000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:19:16.071 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:15.905+0000 7f8f0f123000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:19:16.071 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:16 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:16.069+0000 7f8f0f123000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:19:16.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:16.069+0000 7f64d61d4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:19:16.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:16.153+0000 7f64d61d4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:19:16.439 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:16 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:16.156+0000 7f8f0f123000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:19:17.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:17.166+0000 7f64d61d4000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:19:17.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:17.294+0000 7f64d61d4000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:19:17.525 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:17 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:17.197+0000 7f8f0f123000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:19:17.526 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:17 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:17.329+0000 7f8f0f123000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:19:17.526 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:17 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:17.524+0000 7f8f0f123000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:19:17.939 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:17 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:17.613+0000 7f8f0f123000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:19:17.939 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:17 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:17.752+0000 7f8f0f123000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:19:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:17.482+0000 7f64d61d4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:19:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:17.568+0000 7f64d61d4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:19:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:17.702+0000 7f64d61d4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:19:18.322 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:18.018+0000 7f64d61d4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:19:18.322 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:18.236+0000 7f64d61d4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:19:18.322 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:18.320+0000 7f64d61d4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:19:18.392 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:18 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:18.076+0000 7f8f0f123000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:19:18.392 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:18 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:18.303+0000 7f8f0f123000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:19:18.392 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:18 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:18.390+0000 7f8f0f123000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:19:18.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:18 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:18.565+0000 7f8f0f123000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:19:18.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:18.487+0000 7f64d61d4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:19:19.684 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:19.392+0000 7f64d61d4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:19:19.810 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:19.504+0000 7f8f0f123000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:19:19.943 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:19.682+0000 7f64d61d4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:19:19.943 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:19.776+0000 7f64d61d4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:19:19.943 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:19.924+0000 7f64d61d4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:19:19.943 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: [14/Feb/2024:17:19:19] ENGINE Bus STARTING 2024-02-14T17:19:20.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[24769]: Standby manager daemon x restarted 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[24769]: Standby manager daemon x started 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[29506]: Standby manager daemon x restarted 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:19:20.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:19:20.068 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:19.808+0000 7f8f0f123000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:19:20.068 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:19.907+0000 7f8f0f123000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:19:20.068 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:20.065+0000 7f8f0f123000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:19:20.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: CherryPy Checker: 2024-02-14T17:19:20.304 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: The Application mounted at '' has an empty config. 2024-02-14T17:19:20.304 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: 2024-02-14T17:19:20.304 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: [14/Feb/2024:17:19:20] ENGINE Serving on http://:::9283 2024-02-14T17:19:20.304 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: [14/Feb/2024:17:19:20] ENGINE Bus STARTED 2024-02-14T17:19:20.304 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-mon[25603]: Standby manager daemon x restarted 2024-02-14T17:19:20.304 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-mon[25603]: Standby manager daemon x started 2024-02-14T17:19:20.305 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:19:20.305 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:19:20.305 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:19:20.305 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:19 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3655979649' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:19:21.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[24769]: mgrmap e17: y(active, since 4m), standbys: x 2024-02-14T17:19:21.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[24769]: Active manager daemon y restarted 2024-02-14T17:19:21.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[24769]: Activating manager daemon y 2024-02-14T17:19:21.360 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[24769]: osdmap e59: 8 total, 8 up, 8 in 2024-02-14T17:19:21.361 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:19:21] ENGINE Bus STARTING 2024-02-14T17:19:21.361 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: CherryPy Checker: 2024-02-14T17:19:21.361 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: The Application mounted at '' has an empty config. 2024-02-14T17:19:21.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[29506]: mgrmap e17: y(active, since 4m), standbys: x 2024-02-14T17:19:21.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[29506]: Active manager daemon y restarted 2024-02-14T17:19:21.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[29506]: Activating manager daemon y 2024-02-14T17:19:21.361 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:20 smithi017 ceph-mon[29506]: osdmap e59: 8 total, 8 up, 8 in 2024-02-14T17:19:21.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:20 smithi111 ceph-mon[25603]: mgrmap e17: y(active, since 4m), standbys: x 2024-02-14T17:19:21.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:20 smithi111 ceph-mon[25603]: Active manager daemon y restarted 2024-02-14T17:19:21.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:21 smithi111 ceph-mon[25603]: Activating manager daemon y 2024-02-14T17:19:21.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:21 smithi111 ceph-mon[25603]: osdmap e59: 8 total, 8 up, 8 in 2024-02-14T17:19:21.645 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:19:21.645 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:19:21] ENGINE Serving on http://:::9283 2024-02-14T17:19:21.645 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:19:21] ENGINE Bus STARTED 2024-02-14T17:19:21.645 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:19:21] ENGINE Bus STARTING 2024-02-14T17:19:21.645 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:19:21] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:19:21.645 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:19:21] ENGINE Bus STARTED 2024-02-14T17:19:22.025 INFO:teuthology.orchestra.run.smithi111.stdout:Scheduled node-exporter update... 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: mgrmap e18: y(active, starting, since 0.92913s), standbys: x 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:19:22.320 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:19:22.322 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: Manager daemon y is now available 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:22.323 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: mgrmap e18: y(active, starting, since 0.92913s), standbys: x 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:19:22.324 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: Manager daemon y is now available 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:22.325 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:22.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:22.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:22.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:22.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:22.326 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:22.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:22.326 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: mgrmap e18: y(active, starting, since 0.92913s), standbys: x 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:19:22.338 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: Manager daemon y is now available 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:22.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:22.340 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:19:22.340 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:22.340 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:19:22.340 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:22.687 DEBUG:teuthology.orchestra.run.smithi017:node-exporter.a> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service 2024-02-14T17:19:22.689 DEBUG:teuthology.orchestra.run.smithi111:node-exporter.b> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service 2024-02-14T17:19:22.691 INFO:tasks.cephadm:Adding alertmanager.a on smithi017 2024-02-14T17:19:22.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch apply alertmanager '1;smithi017=a' 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: [14/Feb/2024:17:19:21] ENGINE Bus STARTING 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: [14/Feb/2024:17:19:21] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: [14/Feb/2024:17:19:21] ENGINE Bus STARTED 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: mgrmap e19: y(active, since 1.93296s), standbys: x 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi017=a;smithi111=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: Saving service node-exporter spec with placement smithi017=a;smithi111=b;count:2 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:23.335 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:23 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:23.340 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: [14/Feb/2024:17:19:21] ENGINE Bus STARTING 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: [14/Feb/2024:17:19:21] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: [14/Feb/2024:17:19:21] ENGINE Bus STARTED 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: mgrmap e19: y(active, since 1.93296s), standbys: x 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi017=a;smithi111=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: Saving service node-exporter spec with placement smithi017=a;smithi111=b;count:2 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: [14/Feb/2024:17:19:21] ENGINE Bus STARTING 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: [14/Feb/2024:17:19:21] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:19:23.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: [14/Feb/2024:17:19:21] ENGINE Bus STARTED 2024-02-14T17:19:23.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: mgrmap e19: y(active, since 1.93296s), standbys: x 2024-02-14T17:19:23.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi017=a;smithi111=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:23.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:23.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: Saving service node-exporter spec with placement smithi017=a;smithi111=b;count:2 2024-02-14T17:19:23.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:23.342 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:23 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:24.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[24769]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[24769]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-14T17:19:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:24.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[29506]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:24.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[29506]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-14T17:19:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:24 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:24.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:24 smithi111 ceph-mon[25603]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:24.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:24 smithi111 ceph-mon[25603]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-14T17:19:24.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:24 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:24.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:24 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:24.643 INFO:teuthology.orchestra.run.smithi111.stdout:Scheduled alertmanager update... 2024-02-14T17:19:25.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:25 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:19:25.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:25 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:19:25.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:25 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:25.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:25 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:25.162 DEBUG:teuthology.orchestra.run.smithi017:alertmanager.a> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service 2024-02-14T17:19:25.165 INFO:tasks.cephadm:Adding grafana.a on smithi111 2024-02-14T17:19:25.165 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph orch apply grafana '1;smithi111=a' 2024-02-14T17:19:25.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:19:25.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:19:25.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:25.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:25.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:19:25.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:19:25.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:25.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:25 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.349 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:26 smithi111 ceph-mon[25603]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:26.349 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:26 smithi111 ceph-mon[25603]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-02-14T17:19:26.349 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:26 smithi111 ceph-mon[25603]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:26.349 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:26 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.349 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:26 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.349 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:26 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[24769]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[24769]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[24769]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[29506]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi017=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[29506]: Saving service alertmanager spec with placement smithi017=a;count:1 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[29506]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:26.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:26 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:27.318 INFO:teuthology.orchestra.run.smithi111.stdout:Scheduled grafana update... 2024-02-14T17:19:27.319 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:27 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:19:27.319 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:27 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:19:27.319 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:27 smithi111 ceph-mon[25603]: Deploying daemon prometheus.a on smithi111 2024-02-14T17:19:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:27 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:19:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:27 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:19:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:27 smithi017 ceph-mon[24769]: Deploying daemon prometheus.a on smithi111 2024-02-14T17:19:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:27 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:19:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:27 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:19:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:27 smithi017 ceph-mon[29506]: Deploying daemon prometheus.a on smithi111 2024-02-14T17:19:27.877 DEBUG:teuthology.orchestra.run.smithi111:grafana.a> sudo journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service 2024-02-14T17:19:27.880 INFO:tasks.cephadm:Setting up client nodes... 2024-02-14T17:19:27.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-14T17:19:28.189 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:28 smithi017 ceph-mon[29506]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:28.189 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:28 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:28 smithi017 ceph-mon[24769]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:28 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:28 smithi111 ceph-mon[25603]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:28 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:29.338 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:29 smithi017 ceph-mon[24769]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:29.338 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:29 smithi017 ceph-mon[24769]: Saving service grafana spec with placement smithi111=a;count:1 2024-02-14T17:19:29.338 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:29 smithi017 ceph-mon[29506]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:29.338 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:29 smithi017 ceph-mon[29506]: Saving service grafana spec with placement smithi111=a;count:1 2024-02-14T17:19:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:29 smithi111 ceph-mon[25603]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi111=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:19:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:29 smithi111 ceph-mon[25603]: Saving service grafana spec with placement smithi111=a;count:1 2024-02-14T17:19:29.835 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2024-02-14T17:19:29.835 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQAh9sxliaGGMRAAJu7FFoWAINjGogye3v7WUw== 2024-02-14T17:19:30.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:30 smithi017 ceph-mon[24769]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:30.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1456789958' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:30.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1456789958' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:19:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:30 smithi017 ceph-mon[29506]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1456789958' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1456789958' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:19:30.474 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-02-14T17:19:30.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-14T17:19:30.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-14T17:19:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:30 smithi111 ceph-mon[25603]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1456789958' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1456789958' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:19:30.517 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-14T17:19:32.129 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:32 smithi111 ceph-mon[25603]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:32.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:32 smithi017 ceph-mon[24769]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:32 smithi017 ceph-mon[29506]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:32.891 INFO:teuthology.orchestra.run.smithi111.stdout:[client.1] 2024-02-14T17:19:32.891 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQAk9sxlAUjqNBAA7GElZ+MBcP6igjhuQ+vJtA== 2024-02-14T17:19:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:33 smithi017 ceph-mon[24769]: from='client.? 172.21.15.111:0/234376821' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:33 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:33 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:19:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:33 smithi017 ceph-mon[29506]: from='client.? 172.21.15.111:0/234376821' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:33 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:33 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:19:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:33 smithi111 ceph-mon[25603]: from='client.? 172.21.15.111:0/234376821' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:33 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-14T17:19:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:33 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-14T17:19:33.576 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-14T17:19:33.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-14T17:19:33.576 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-14T17:19:33.617 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-14T17:19:33.617 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-14T17:19:33.617 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph mgr dump --format=json 2024-02-14T17:19:33.834 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:34.238 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:34 smithi017 ceph-mon[24769]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:34.239 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:34 smithi017 ceph-mon[29506]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:34 smithi111 ceph-mon[25603]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:35.728 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:19:36.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:35 smithi017 ceph-mon[29506]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:36.332 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2393793638' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:19:36.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:35 smithi017 ceph-mon[24769]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:36.332 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:35 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2393793638' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:19:36.353 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":20,"active_gid":24340,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":3428894855},{"type":"v1","addr":"172.21.15.17:6801","nonce":3428894855}]},"active_addr":"172.21.15.17:6801/3428894855","active_change":"2024-02-14T17:19:20.079952+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24332,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.17:8443/","prometheus":"http://172.21.15.17:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":4251098973}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3178045652}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":1369078824}]},{"addrvec":[{"type":"v2","addr":"172.21.15.17:0","nonce":3842588073}]}]}} 2024-02-14T17:19:36.356 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-14T17:19:36.357 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-14T17:19:36.357 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-14T17:19:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:35 smithi111 ceph-mon[25603]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:35 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2393793638' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:19:36.573 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:38.145 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:19:38.146 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":59,"fsid":"612435b4-cb5c-11ee-95b9-87774f69a715","created":"2024-02-14T17:14:10.781716+0000","modified":"2024-02-14T17:19:20.079644+0000","last_up_change":"2024-02-14T17:19:01.887456+0000","last_in_change":"2024-02-14T17:18:44.869711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-14T17:17:10.259165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a980d777-c667-4148-b5fa-b9fc89d6dc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6803","nonce":2126863272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6805","nonce":2126863272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6809","nonce":2126863272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6807","nonce":2126863272}]},"public_addr":"172.21.15.17:6803/2126863272","cluster_addr":"172.21.15.17:6805/2126863272","heartbeat_back_addr":"172.21.15.17:6809/2126863272","heartbeat_front_addr":"172.21.15.17:6807/2126863272","state":["exists","up"]},{"osd":1,"uuid":"517877bd-896e-4e1d-93e6-2a704bca2e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6811","nonce":336965104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6813","nonce":336965104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6817","nonce":336965104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6815","nonce":336965104}]},"public_addr":"172.21.15.17:6811/336965104","cluster_addr":"172.21.15.17:6813/336965104","heartbeat_back_addr":"172.21.15.17:6817/336965104","heartbeat_front_addr":"172.21.15.17:6815/336965104","state":["exists","up"]},{"osd":2,"uuid":"c810adb5-4205-4e6a-a102-8cdeafeb7964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6819","nonce":3232569458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6821","nonce":3232569458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6825","nonce":3232569458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6823","nonce":3232569458}]},"public_addr":"172.21.15.17:6819/3232569458","cluster_addr":"172.21.15.17:6821/3232569458","heartbeat_back_addr":"172.21.15.17:6825/3232569458","heartbeat_front_addr":"172.21.15.17:6823/3232569458","state":["exists","up"]},{"osd":3,"uuid":"0c10a4f3-4cd7-44da-8859-96c78d7b5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6827","nonce":2789642823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6829","nonce":2789642823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6833","nonce":2789642823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6831","nonce":2789642823}]},"public_addr":"172.21.15.17:6827/2789642823","cluster_addr":"172.21.15.17:6829/2789642823","heartbeat_back_addr":"172.21.15.17:6833/2789642823","heartbeat_front_addr":"172.21.15.17:6831/2789642823","state":["exists","up"]},{"osd":4,"uuid":"82f83614-981c-4b5f-bdba-00241f18bbb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6801","nonce":2508759102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6803","nonce":2508759102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6807","nonce":2508759102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6805","nonce":2508759102}]},"public_addr":"172.21.15.111:6801/2508759102","cluster_addr":"172.21.15.111:6803/2508759102","heartbeat_back_addr":"172.21.15.111:6807/2508759102","heartbeat_front_addr":"172.21.15.111:6805/2508759102","state":["exists","up"]},{"osd":5,"uuid":"6c46f35f-fb44-46f9-a078-20ff42d28ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6809","nonce":441844094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6811","nonce":441844094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6815","nonce":441844094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6813","nonce":441844094}]},"public_addr":"172.21.15.111:6809/441844094","cluster_addr":"172.21.15.111:6811/441844094","heartbeat_back_addr":"172.21.15.111:6815/441844094","heartbeat_front_addr":"172.21.15.111:6813/441844094","state":["exists","up"]},{"osd":6,"uuid":"3c5091bd-4021-4c53-91d3-5a9bdebb85b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6817","nonce":3964813343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6819","nonce":3964813343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6823","nonce":3964813343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6821","nonce":3964813343}]},"public_addr":"172.21.15.111:6817/3964813343","cluster_addr":"172.21.15.111:6819/3964813343","heartbeat_back_addr":"172.21.15.111:6823/3964813343","heartbeat_front_addr":"172.21.15.111:6821/3964813343","state":["exists","up"]},{"osd":7,"uuid":"a9cc83c8-5fd6-4e76-92df-c7c15b15b86d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6825","nonce":661554482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6827","nonce":661554482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6831","nonce":661554482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6829","nonce":661554482}]},"public_addr":"172.21.15.111:6825/661554482","cluster_addr":"172.21.15.111:6827/661554482","heartbeat_back_addr":"172.21.15.111:6831/661554482","heartbeat_front_addr":"172.21.15.111:6829/661554482","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:16:22.794805+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:16:45.311188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:07.896583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:30.997825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:52.265643+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:18:14.142700+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:18:36.986785+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:19:00.036720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2485983711":"2024-02-15T17:19:20.079598+0000","172.21.15.17:6801/1844808215":"2024-02-15T17:19:20.079598+0000","172.21.15.17:0/2684050458":"2024-02-15T17:19:20.079598+0000","172.21.15.17:0/2689455788":"2024-02-15T17:15:05.331154+0000","172.21.15.17:6800/1844808215":"2024-02-15T17:19:20.079598+0000","172.21.15.17:0/1905974843":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/996094750":"2024-02-15T17:14:42.920120+0000","172.21.15.17:0/1219630449":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/1994288429":"2024-02-15T17:14:42.920120+0000","172.21.15.17:0/316814259":"2024-02-15T17:19:20.079598+0000","172.21.15.17:6800/1397676400":"2024-02-15T17:14:42.920120+0000","172.21.15.17:0/3513237918":"2024-02-15T17:19:20.079598+0000","172.21.15.17:6801/1666345519":"2024-02-15T17:15:05.331154+0000","172.21.15.17:6801/1397676400":"2024-02-15T17:14:42.920120+0000","172.21.15.17:6800/1666345519":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/808657630":"2024-02-15T17:14:42.920120+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-14T17:19:38.435 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:38 smithi017 ceph-mon[24769]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:38.436 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:38 smithi017 ceph-mon[29506]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:38 smithi111 ceph-mon[25603]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:38.743 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-14T17:19:38.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd dump --format=json 2024-02-14T17:19:38.961 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:39.163 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:39 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1881033750' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:39.330 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:39 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1881033750' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:39.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:39 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1881033750' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:40.114 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:40 smithi017 ceph-mon[24769]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:40.115 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:40 smithi017 ceph-mon[29506]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:40 smithi111 ceph-mon[25603]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:40.514 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:19:40.514 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":59,"fsid":"612435b4-cb5c-11ee-95b9-87774f69a715","created":"2024-02-14T17:14:10.781716+0000","modified":"2024-02-14T17:19:20.079644+0000","last_up_change":"2024-02-14T17:19:01.887456+0000","last_in_change":"2024-02-14T17:18:44.869711+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-14T17:17:10.259165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a980d777-c667-4148-b5fa-b9fc89d6dc45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6803","nonce":2126863272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6805","nonce":2126863272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6809","nonce":2126863272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":2126863272},{"type":"v1","addr":"172.21.15.17:6807","nonce":2126863272}]},"public_addr":"172.21.15.17:6803/2126863272","cluster_addr":"172.21.15.17:6805/2126863272","heartbeat_back_addr":"172.21.15.17:6809/2126863272","heartbeat_front_addr":"172.21.15.17:6807/2126863272","state":["exists","up"]},{"osd":1,"uuid":"517877bd-896e-4e1d-93e6-2a704bca2e77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6811","nonce":336965104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6813","nonce":336965104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6817","nonce":336965104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":336965104},{"type":"v1","addr":"172.21.15.17:6815","nonce":336965104}]},"public_addr":"172.21.15.17:6811/336965104","cluster_addr":"172.21.15.17:6813/336965104","heartbeat_back_addr":"172.21.15.17:6817/336965104","heartbeat_front_addr":"172.21.15.17:6815/336965104","state":["exists","up"]},{"osd":2,"uuid":"c810adb5-4205-4e6a-a102-8cdeafeb7964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6819","nonce":3232569458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6821","nonce":3232569458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6824","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6825","nonce":3232569458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":3232569458},{"type":"v1","addr":"172.21.15.17:6823","nonce":3232569458}]},"public_addr":"172.21.15.17:6819/3232569458","cluster_addr":"172.21.15.17:6821/3232569458","heartbeat_back_addr":"172.21.15.17:6825/3232569458","heartbeat_front_addr":"172.21.15.17:6823/3232569458","state":["exists","up"]},{"osd":3,"uuid":"0c10a4f3-4cd7-44da-8859-96c78d7b5da0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6826","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6827","nonce":2789642823}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6828","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6829","nonce":2789642823}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6832","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6833","nonce":2789642823}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6830","nonce":2789642823},{"type":"v1","addr":"172.21.15.17:6831","nonce":2789642823}]},"public_addr":"172.21.15.17:6827/2789642823","cluster_addr":"172.21.15.17:6829/2789642823","heartbeat_back_addr":"172.21.15.17:6833/2789642823","heartbeat_front_addr":"172.21.15.17:6831/2789642823","state":["exists","up"]},{"osd":4,"uuid":"82f83614-981c-4b5f-bdba-00241f18bbb9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6801","nonce":2508759102}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6803","nonce":2508759102}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6807","nonce":2508759102}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2508759102},{"type":"v1","addr":"172.21.15.111:6805","nonce":2508759102}]},"public_addr":"172.21.15.111:6801/2508759102","cluster_addr":"172.21.15.111:6803/2508759102","heartbeat_back_addr":"172.21.15.111:6807/2508759102","heartbeat_front_addr":"172.21.15.111:6805/2508759102","state":["exists","up"]},{"osd":5,"uuid":"6c46f35f-fb44-46f9-a078-20ff42d28ce8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6809","nonce":441844094}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6811","nonce":441844094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6815","nonce":441844094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":441844094},{"type":"v1","addr":"172.21.15.111:6813","nonce":441844094}]},"public_addr":"172.21.15.111:6809/441844094","cluster_addr":"172.21.15.111:6811/441844094","heartbeat_back_addr":"172.21.15.111:6815/441844094","heartbeat_front_addr":"172.21.15.111:6813/441844094","state":["exists","up"]},{"osd":6,"uuid":"3c5091bd-4021-4c53-91d3-5a9bdebb85b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6817","nonce":3964813343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6819","nonce":3964813343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6823","nonce":3964813343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":3964813343},{"type":"v1","addr":"172.21.15.111:6821","nonce":3964813343}]},"public_addr":"172.21.15.111:6817/3964813343","cluster_addr":"172.21.15.111:6819/3964813343","heartbeat_back_addr":"172.21.15.111:6823/3964813343","heartbeat_front_addr":"172.21.15.111:6821/3964813343","state":["exists","up"]},{"osd":7,"uuid":"a9cc83c8-5fd6-4e76-92df-c7c15b15b86d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6825","nonce":661554482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6827","nonce":661554482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6831","nonce":661554482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":661554482},{"type":"v1","addr":"172.21.15.111:6829","nonce":661554482}]},"public_addr":"172.21.15.111:6825/661554482","cluster_addr":"172.21.15.111:6827/661554482","heartbeat_back_addr":"172.21.15.111:6831/661554482","heartbeat_front_addr":"172.21.15.111:6829/661554482","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:16:22.794805+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:16:45.311188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:07.896583+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:30.997825+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:17:52.265643+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:18:14.142700+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:18:36.986785+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-02-14T17:19:00.036720+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.17:0/2485983711":"2024-02-15T17:19:20.079598+0000","172.21.15.17:6801/1844808215":"2024-02-15T17:19:20.079598+0000","172.21.15.17:0/2684050458":"2024-02-15T17:19:20.079598+0000","172.21.15.17:0/2689455788":"2024-02-15T17:15:05.331154+0000","172.21.15.17:6800/1844808215":"2024-02-15T17:19:20.079598+0000","172.21.15.17:0/1905974843":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/996094750":"2024-02-15T17:14:42.920120+0000","172.21.15.17:0/1219630449":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/1994288429":"2024-02-15T17:14:42.920120+0000","172.21.15.17:0/316814259":"2024-02-15T17:19:20.079598+0000","172.21.15.17:6800/1397676400":"2024-02-15T17:14:42.920120+0000","172.21.15.17:0/3513237918":"2024-02-15T17:19:20.079598+0000","172.21.15.17:6801/1666345519":"2024-02-15T17:15:05.331154+0000","172.21.15.17:6801/1397676400":"2024-02-15T17:14:42.920120+0000","172.21.15.17:6800/1666345519":"2024-02-15T17:15:05.331154+0000","172.21.15.17:0/808657630":"2024-02-15T17:14:42.920120+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-14T17:19:41.073 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-14T17:19:41.073 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-14T17:19:41.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-14T17:19:41.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-14T17:19:41.074 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-14T17:19:41.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-14T17:19:41.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-14T17:19:41.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-14T17:19:41.079 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:40 smithi111 systemd[1]: Starting Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:19:41.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1273993746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:41.339 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 podman[40725]: 2024-02-14 17:19:41.059251361 +0000 UTC m=+0.018942420 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-14T17:19:41.339 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 podman[40725]: 2024-02-14 17:19:41.216451765 +0000 UTC m=+0.176142815 container create 91b366745ac6fe83d44ac072578c64f8adff443679125e0f08523d3e9989b1b8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:19:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:41 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1273993746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:41.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:41 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1273993746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-14T17:19:41.564 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.567 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.567 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.567 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.570 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.570 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.575 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.580 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 podman[40725]: 2024-02-14 17:19:41.441005301 +0000 UTC m=+0.400696350 container init 91b366745ac6fe83d44ac072578c64f8adff443679125e0f08523d3e9989b1b8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 podman[40725]: 2024-02-14 17:19:41.443416431 +0000 UTC m=+0.403107481 container start 91b366745ac6fe83d44ac072578c64f8adff443679125e0f08523d3e9989b1b8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.466Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.466Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.466Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.466Z caller=main.go:518 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi111 (none))" 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.466Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.466Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.467Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:19:41.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.467Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.468Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.471Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.471Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.739µs 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.471Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.471Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.471Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.334µs wal_replay_duration=143.758µs total_replay_duration=179.222µs 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.472Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.472Z caller=main.go:947 level=info msg="TSDB started" 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.472Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.487Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.080477ms db_storage=688ns remote_storage=1.2µs web_handler=392ns query_engine=770ns scrape=363.333µs scrape_sd=16.659µs notify=861ns notify_sd=1.726µs rules=14.52673ms 2024-02-14T17:19:41.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:19:41.487Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-14T17:19:41.737 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 bash[40725]: 91b366745ac6fe83d44ac072578c64f8adff443679125e0f08523d3e9989b1b8 2024-02-14T17:19:41.737 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:19:41 smithi111 systemd[1]: Started Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:19:42.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:42 smithi017 ceph-mon[24769]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:42.306 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:42 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:42.306 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:42 smithi017 ceph-mon[29506]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:42.306 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:42 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:42 smithi111 ceph-mon[25603]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:42 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:43.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:43 smithi017 ceph-mon[24769]: Deploying daemon node-exporter.a on smithi017 2024-02-14T17:19:43.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:43 smithi017 ceph-mon[29506]: Deploying daemon node-exporter.a on smithi017 2024-02-14T17:19:43.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:43 smithi111 ceph-mon[25603]: Deploying daemon node-exporter.a on smithi017 2024-02-14T17:19:44.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:44 smithi017 ceph-mon[24769]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:44 smithi017 ceph-mon[29506]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:44 smithi111 ceph-mon[25603]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:45 smithi017 ceph-mon[24769]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:46 smithi017 ceph-mon[29506]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:46.286 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:46 smithi111 ceph-mon[25603]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:47 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:47.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:47 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:47.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:47 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:19:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:19:48.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:48 smithi017 ceph-mon[29506]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:48 smithi017 ceph-mon[24769]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:48.428 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:48 smithi111 ceph-mon[25603]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:50.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:50 smithi017 ceph-mon[24769]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:50.313 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:50 smithi017 ceph-mon[29506]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:50.389 INFO:teuthology.orchestra.run.smithi017.stdout:197568495632 2024-02-14T17:19:50.389 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-14T17:19:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:50 smithi111 ceph-mon[25603]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:51.416 INFO:teuthology.orchestra.run.smithi017.stdout:231928233995 2024-02-14T17:19:51.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-14T17:19:52.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:52 smithi017 ceph-mon[24769]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:52.340 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:52 smithi017 ceph-mon[29506]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:52 smithi111 ceph-mon[25603]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:52.839 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:52.840 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:19:54.351 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:54 smithi017 ceph-mon[24769]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:54.351 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:54 smithi017 ceph-mon[29506]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:54.352 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:19:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:19:54] "GET /metrics HTTP/1.1" 200 191486 "" "Prometheus/2.33.4" 2024-02-14T17:19:54.352 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:54 smithi017 bash[45563]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-14T17:19:54.352 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:54 smithi017 bash[45563]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-14T17:19:54.352 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:54 smithi017 bash[45563]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-14T17:19:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:54 smithi111 ceph-mon[25603]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:54.901 INFO:teuthology.orchestra.run.smithi017.stdout:163208757269 2024-02-14T17:19:54.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-14T17:19:56.378 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:56 smithi017 ceph-mon[24769]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:56.378 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:56 smithi017 ceph-mon[29506]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:56 smithi111 ceph-mon[25603]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:56.689 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:56 smithi017 bash[45563]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-14T17:19:57.189 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:56 smithi017 bash[45563]: Writing manifest to image destination 2024-02-14T17:19:57.241 INFO:teuthology.orchestra.run.smithi017.stdout:128849018905 2024-02-14T17:19:57.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-14T17:19:57.553 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:57 smithi017 podman[45563]: 2024-02-14 17:19:57.26140737 +0000 UTC m=+5.381384306 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-14T17:19:57.580 INFO:teuthology.orchestra.run.smithi017.stdout:107374182430 2024-02-14T17:19:57.581 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-14T17:19:57.933 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:19:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:19:58.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:19:58 smithi017 ceph-mon[24769]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:58.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:19:58 smithi017 ceph-mon[29506]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:19:58 smithi111 ceph-mon[25603]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:19:58.809 INFO:teuthology.orchestra.run.smithi017.stdout:55834574887 2024-02-14T17:19:58.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-14T17:19:59.689 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 podman[45563]: 2024-02-14 17:19:59.40452295 +0000 UTC m=+7.524499878 container create dfc53c1158e99a6eb829a1f3159da33140058102d8058c98dce862d782d0d869 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 podman[45563]: 2024-02-14 17:19:59.809492084 +0000 UTC m=+7.929469012 container init dfc53c1158e99a6eb829a1f3159da33140058102d8058c98dce862d782d0d869 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 podman[45563]: 2024-02-14 17:19:59.812631003 +0000 UTC m=+7.932607942 container start dfc53c1158e99a6eb829a1f3159da33140058102d8058c98dce862d782d0d869 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.816Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.816Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.817Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.817Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.817Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=arp 2024-02-14T17:20:00.076 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-14T17:20:00.077 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=edac 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-14T17:20:00.078 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=os 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=stat 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-14T17:20:00.079 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=time 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=uname 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[46058]: ts=2024-02-14T17:19:59.818Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 bash[45563]: dfc53c1158e99a6eb829a1f3159da33140058102d8058c98dce862d782d0d869 2024-02-14T17:20:00.080 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:19:59 smithi017 systemd[1]: Started Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:00.266 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:00 smithi111 ceph-mon[25603]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:00.266 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:00 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:00.266 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:00 smithi111 ceph-mon[25603]: overall HEALTH_OK 2024-02-14T17:20:00.339 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:00 smithi017 ceph-mon[24769]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:00.339 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:00 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:00.339 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:00 smithi017 ceph-mon[24769]: overall HEALTH_OK 2024-02-14T17:20:00.339 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:00 smithi017 ceph-mon[29506]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:00.339 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:00 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:00.339 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:00 smithi017 ceph-mon[29506]: overall HEALTH_OK 2024-02-14T17:20:00.607 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:00 smithi111 systemd[1]: Starting Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:20:00.779 INFO:teuthology.orchestra.run.smithi017.stdout:34359738412 2024-02-14T17:20:00.780 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-14T17:20:00.984 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:00 smithi111 bash[40978]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-02-14T17:20:01.353 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:01 smithi017 ceph-mon[29506]: Deploying daemon node-exporter.b on smithi111 2024-02-14T17:20:01.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:01 smithi017 ceph-mon[24769]: Deploying daemon node-exporter.b on smithi111 2024-02-14T17:20:01.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:01 smithi111 ceph-mon[25603]: Deploying daemon node-exporter.b on smithi111 2024-02-14T17:20:01.484 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:01 smithi111 bash[40978]: Getting image source signatures 2024-02-14T17:20:01.484 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:01 smithi111 bash[40978]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-02-14T17:20:01.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:01 smithi111 bash[40978]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-02-14T17:20:01.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:01 smithi111 bash[40978]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-02-14T17:20:01.756 INFO:teuthology.orchestra.run.smithi017.stdout:197568495634 2024-02-14T17:20:02.179 INFO:teuthology.orchestra.run.smithi017.stdout:77309411363 2024-02-14T17:20:02.180 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-14T17:20:02.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:02 smithi017 ceph-mon[24769]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:02.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:02 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3531318244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-14T17:20:02.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:02 smithi017 ceph-mon[29506]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:02.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:02 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3531318244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-14T17:20:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:02 smithi111 ceph-mon[25603]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:02 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3531318244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-14T17:20:02.993 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495632 got 197568495634 for osd.6 2024-02-14T17:20:02.994 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:03.045 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:03.047 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:03.049 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:03.049 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:03.353 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:03.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:03 smithi111 bash[40978]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-02-14T17:20:03.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:03 smithi111 bash[40978]: Writing manifest to image destination 2024-02-14T17:20:04.082 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:03 smithi111 podman[40978]: 2024-02-14 17:20:03.793731628 +0000 UTC m=+3.088362183 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-14T17:20:04.083 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:03 smithi111 podman[40978]: 2024-02-14 17:20:03.875803552 +0000 UTC m=+3.170434108 container create 8859571f4a9e85cbd8b2b6bb9f9e9d9b32058bbc7639e966b784dc78558be397 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:20:04.305 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:04 smithi017 ceph-mon[24769]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:04.305 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:04 smithi017 ceph-mon[29506]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:04.306 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:20:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:04] "GET /metrics HTTP/1.1" 200 191486 "" "Prometheus/2.33.4" 2024-02-14T17:20:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-mon[25603]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:04.484 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 podman[40978]: 2024-02-14 17:20:04.119631929 +0000 UTC m=+3.414262484 container init 8859571f4a9e85cbd8b2b6bb9f9e9d9b32058bbc7639e966b784dc78558be397 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 podman[40978]: 2024-02-14 17:20:04.122799868 +0000 UTC m=+3.417430431 container start 8859571f4a9e85cbd8b2b6bb9f9e9d9b32058bbc7639e966b784dc78558be397 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.127Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.127Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.127Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.127Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=arp 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-14T17:20:04.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=edac 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-14T17:20:04.486 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=os 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-14T17:20:04.487 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=stat 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=time 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=uname 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-14T17:20:04.488 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[41029]: ts=2024-02-14T17:20:04.128Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-14T17:20:04.489 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 bash[40978]: 8859571f4a9e85cbd8b2b6bb9f9e9d9b32058bbc7639e966b784dc78558be397 2024-02-14T17:20:04.489 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:20:04 smithi111 systemd[1]: Started Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:05.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:05 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:05.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:05 smithi017 ceph-mon[29506]: Deploying daemon alertmanager.a on smithi017 2024-02-14T17:20:05.588 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:05 smithi017 ceph-mon[29506]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:05.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:05 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:05.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:05 smithi017 ceph-mon[24769]: Deploying daemon alertmanager.a on smithi017 2024-02-14T17:20:05.589 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:05 smithi017 ceph-mon[24769]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:05 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:05 smithi111 ceph-mon[25603]: Deploying daemon alertmanager.a on smithi017 2024-02-14T17:20:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:05 smithi111 ceph-mon[25603]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:07.078 INFO:teuthology.orchestra.run.smithi017.stdout:77309411365 2024-02-14T17:20:07.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:07 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:07.359 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:07 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:07 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:20:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:20:08.358 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:08 smithi017 ceph-mon[29506]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:08.358 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:08 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2018150173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-14T17:20:08.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:08 smithi017 ceph-mon[24769]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:08.358 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:08 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2018150173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-14T17:20:08.429 INFO:teuthology.orchestra.run.smithi017.stdout:231928233999 2024-02-14T17:20:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:08 smithi111 ceph-mon[25603]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:08 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2018150173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-14T17:20:08.740 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411365 for osd.2 2024-02-14T17:20:08.741 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:09.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:09 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/402367735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-14T17:20:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:09 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/402367735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-14T17:20:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:09 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/402367735' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-14T17:20:10.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:10 smithi017 ceph-mon[24769]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:10.369 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:10 smithi017 ceph-mon[29506]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:10 smithi111 ceph-mon[25603]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:11.912 INFO:teuthology.orchestra.run.smithi017.stdout:34359738415 2024-02-14T17:20:12.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:12 smithi017 ceph-mon[24769]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:12.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:12 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1148902325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-14T17:20:12.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:12 smithi017 ceph-mon[29506]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:12.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:12 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1148902325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-14T17:20:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:12 smithi111 ceph-mon[25603]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:12 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1148902325' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-14T17:20:12.607 INFO:teuthology.orchestra.run.smithi017.stdout:107374182433 2024-02-14T17:20:13.435 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:13 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4155033301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-14T17:20:13.435 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:13 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4155033301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-14T17:20:13.458 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233995 got 231928233999 for osd.7 2024-02-14T17:20:13.458 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:13.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:13 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4155033301' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-14T17:20:14.028 INFO:teuthology.orchestra.run.smithi017.stdout:163208757274 2024-02-14T17:20:14.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:14 smithi017 ceph-mon[24769]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:14.362 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:14 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2002098797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-14T17:20:14.362 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:14 smithi017 ceph-mon[29506]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:14.362 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:14 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2002098797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-14T17:20:14.363 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:20:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:14] "GET /metrics HTTP/1.1" 200 191469 "" "Prometheus/2.33.4" 2024-02-14T17:20:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:14 smithi111 ceph-mon[25603]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:14 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2002098797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-14T17:20:16.082 INFO:teuthology.orchestra.run.smithi017.stdout:55834574891 2024-02-14T17:20:16.083 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:16 smithi017 ceph-mon[24769]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:16.083 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:16 smithi017 ceph-mon[29506]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:16.286 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:16 smithi111 ceph-mon[25603]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:17.150 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:17 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/72842304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-14T17:20:17.151 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:17 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/72842304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-14T17:20:17.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738412 got 34359738415 for osd.0 2024-02-14T17:20:17.365 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:17.438 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:17 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/72842304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-14T17:20:17.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:20:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:20:18.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:18 smithi017 ceph-mon[24769]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:18.370 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:18 smithi017 ceph-mon[29506]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:18 smithi111 ceph-mon[25603]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:18.733 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574887 got 55834574891 for osd.1 2024-02-14T17:20:18.734 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:19.648 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:19 smithi017 systemd[1]: Starting Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:20:19.675 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182433 for osd.3 2024-02-14T17:20:19.675 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:20.032 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757269 got 163208757274 for osd.5 2024-02-14T17:20:20.032 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:20.427 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-mon[24769]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:20.428 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-mon[29506]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:20.429 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 podman[47210]: 2024-02-14 17:20:20.17084372 +0000 UTC m=+0.109021234 volume create 47f06caf10e53e46cc46ce9f9d2e518b7e026b7675e40b75ffe01c1bcee2eb2f 2024-02-14T17:20:20.429 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 podman[47210]: 2024-02-14 17:20:20.080424827 +0000 UTC m=+0.018602343 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-14T17:20:20.429 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 podman[47210]: 2024-02-14 17:20:20.246282324 +0000 UTC m=+0.184459837 container create 11fd55bbf57f7a9db34a6a20e47617906028407ed1be0633ab0fa5c050a28831 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:20 smithi111 ceph-mon[25603]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 podman[47210]: 2024-02-14 17:20:20.548856923 +0000 UTC m=+0.487034439 container init 11fd55bbf57f7a9db34a6a20e47617906028407ed1be0633ab0fa5c050a28831 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 podman[47210]: 2024-02-14 17:20:20.551878102 +0000 UTC m=+0.490055624 container start 11fd55bbf57f7a9db34a6a20e47617906028407ed1be0633ab0fa5c050a28831 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.561Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.561Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.563Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.564Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.589Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:20:20.843 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.590Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:20:20.844 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.592Z caller=main.go:518 msg=Listening address=:9093 2024-02-14T17:20:20.844 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:20.592Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-14T17:20:20.844 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 bash[47210]: 11fd55bbf57f7a9db34a6a20e47617906028407ed1be0633ab0fa5c050a28831 2024-02-14T17:20:20.844 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:20 smithi017 systemd[1]: Started Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:21.378 INFO:teuthology.orchestra.run.smithi017.stdout:128849018910 2024-02-14T17:20:21.924 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:21.924 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:21.924 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:21.924 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:21.924 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:21.924 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/727688214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:21.925 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:20:21.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:20:21.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:20:21.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:20:21.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:21 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/727688214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:20:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:20:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:20:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:21 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/727688214' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-14T17:20:22.320 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018905 got 128849018910 for osd.4 2024-02-14T17:20:22.320 DEBUG:teuthology.parallel:result is None 2024-02-14T17:20:22.320 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-14T17:20:22.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-14T17:20:22.681 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:22.894 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:22 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:22.565Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00098896s 2024-02-14T17:20:24.185 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:24 smithi017 ceph-mon[24769]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:24.185 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:24 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:24.185 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:24 smithi017 ceph-mon[29506]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:24.185 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:24 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:20:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:24] "GET /metrics HTTP/1.1" 200 191470 "" "Prometheus/2.33.4" 2024-02-14T17:20:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:24 smithi111 ceph-mon[25603]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:24 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:24.852 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:20:24.854 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-02-14T17:20:26.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:26 smithi017 ceph-mon[24769]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:20:26.168 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:26 smithi017 ceph-mon[24769]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:26.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:26 smithi017 ceph-mon[29506]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:20:26.168 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:26 smithi017 ceph-mon[29506]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:26.196 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-02-14T17:20:23.023427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48884,"kb_used_data":6068,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912332,"statfs":{"total":767960285184,"available":767910227968,"internally_reserved":0,"allocated":6213632,"data_stored":3486615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002204"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-14T17:19:21.105967+0000","last_change":"2024-02-14T17:19:05.912448+0000","last_active":"2024-02-14T17:19:21.105967+0000","last_peered":"2024-02-14T17:19:21.105967+0000","last_clean":"2024-02-14T17:19:21.105967+0000","last_became_active":"2024-02-14T17:19:05.911793+0000","last_became_peered":"2024-02-14T17:19:05.911793+0000","last_unstale":"2024-02-14T17:19:21.105967+0000","last_undegraded":"2024-02-14T17:19:21.105967+0000","last_fullsized":"2024-02-14T17:19:21.105967+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:17:10.961265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:17:10.961265+0000","last_clean_scrub_stamp":"2024-02-14T17:17:10.961265+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:35:14.263803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.59799999999999998}]},{"osd":1,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.56499999999999995}]},{"osd":2,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.625}]},{"osd":3,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.58199999999999996}]},{"osd":4,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.39300000000000002}]},{"osd":5,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.53200000000000003}]},{"osd":6,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.47599999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.51300000000000001}]},{"osd":1,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.432}]},{"osd":3,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55400000000000005}]},{"osd":4,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.49099999999999999}]},{"osd":5,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.58999999999999997}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.64900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33700000000000002,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.36499999999999999,"15min":0.33900000000000002},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57499999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.372}]},{"osd":2,"last update":"Wed Feb 14 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.311,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.42199999999999999,"15min":0.41299999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54400000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.39400000000000002}]},{"osd":3,"last update":"Wed Feb 14 17:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.35099999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60699999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.54000000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.38300000000000001}]},{"osd":4,"last update":"Wed Feb 14 17:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.5}]},{"osd":5,"last update":"Wed Feb 14 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.625,"15min":0.621},"min":{"1min":0.40000000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83899999999999997,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.47099999999999997}]},{"osd":6,"last update":"Wed Feb 14 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.434},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Wed Feb 14 17:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.443},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.45800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 14 17:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.35299999999999998,"15min":0.33400000000000002},"min":{"1min":0.28399999999999997,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.55300000000000005,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.36599999999999999,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.50600000000000001,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Feb 14 17:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.41099999999999998},"min":{"1min":0.25,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.47199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57699999999999996,"5min":0.624,"15min":0.624},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.40200000000000002}]},{"osd":4,"last update":"Wed Feb 14 17:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78000000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.60899999999999999}]},{"osd":5,"last update":"Wed Feb 14 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79600000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63,"15min":0.63300000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77300000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.56499999999999995}]},{"osd":6,"last update":"Wed Feb 14 17:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.46899999999999997}]},{"osd":7,"last update":"Wed Feb 14 17:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.54200000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6264,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738888,"statfs":{"total":95995035648,"available":95988621312,"internally_reserved":0,"allocated":573440,"data_stored":236915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56899999999999995,"5min":0.628,"15min":0.628},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.30299999999999999}]},{"osd":1,"last update":"Wed Feb 14 17:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.27500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60099999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.63200000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.34000000000000002}]},{"osd":3,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.53900000000000003,"5min":0.751,"15min":0.751},"last":0.371}]},{"osd":4,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.437}]},{"osd":5,"last update":"Wed Feb 14 17:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42699999999999999}]},{"osd":6,"last update":"Wed Feb 14 17:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.52300000000000002}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.54700000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5688,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739464,"statfs":{"total":95995035648,"available":95989211136,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.40500000000000003,"15min":0.40000000000000002},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.438,"15min":0.439},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.54500000000000004}]},{"osd":1,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.439,"15min":0.439},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.5},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.434},"min":{"1min":0.32200000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65600000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.60499999999999998}]},{"osd":2,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.47499999999999998,"15min":0.46899999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.52200000000000002}]},{"osd":4,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57699999999999996,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.63},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.877,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.627,"15min":0.63},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.751,"15min":0.65000000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":14.73,"5min":14.73,"15min":14.73},"last":0.622}]},{"osd":6,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.66800000000000004}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.66300000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52000000000000002,"15min":0.51800000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.629},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72999999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.50700000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85099999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.68999999999999995}]},{"osd":2,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70999999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.70199999999999996}]},{"osd":3,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.33900000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76500000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70299999999999996,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.65300000000000002}]},{"osd":5,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.38800000000000001,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.29399999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.438}]},{"osd":6,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.53300000000000003}]},{"osd":7,"last update":"Wed Feb 14 17:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.496},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54800000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5688,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739464,"statfs":{"total":95995035648,"available":95989211136,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53300000000000003,"15min":0.53700000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.72299999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.42999999999999999}]},{"osd":2,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53700000000000003,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66300000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81200000000000006,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.36799999999999999}]},{"osd":3,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.64700000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.38},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.38500000000000001}]},{"osd":4,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40699999999999997,"15min":0.40000000000000002},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65600000000000003,"5min":0.87,"15min":0.87},"last":0.313},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.434,"15min":0.42699999999999999},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63700000000000001,"5min":1.085,"15min":1.085},"last":0.32600000000000001}]},{"osd":6,"last update":"Wed Feb 14 17:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.35199999999999998}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.46000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-14T17:20:26.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph pg dump --format=json 2024-02-14T17:20:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:26 smithi111 ceph-mon[25603]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:20:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:26 smithi111 ceph-mon[25603]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:26.854 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:27.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[24769]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:20:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[29506]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:20:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-mon[25603]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:20:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:28.600 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:20:28.602 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-02-14T17:20:29.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:28 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:20:29.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:28 smithi017 ceph-mon[24769]: Deploying daemon grafana.a on smithi111 2024-02-14T17:20:29.126 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:28 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:20:29.126 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:28 smithi017 ceph-mon[29506]: Deploying daemon grafana.a on smithi111 2024-02-14T17:20:29.127 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-02-14T17:20:27.024344+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48884,"kb_used_data":6068,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912332,"statfs":{"total":767960285184,"available":767910227968,"internally_reserved":0,"allocated":6213632,"data_stored":3486615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002458"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":267,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-02-14T17:19:21.105967+0000","last_change":"2024-02-14T17:19:05.912448+0000","last_active":"2024-02-14T17:19:21.105967+0000","last_peered":"2024-02-14T17:19:21.105967+0000","last_clean":"2024-02-14T17:19:21.105967+0000","last_became_active":"2024-02-14T17:19:05.911793+0000","last_became_peered":"2024-02-14T17:19:05.911793+0000","last_unstale":"2024-02-14T17:19:21.105967+0000","last_undegraded":"2024-02-14T17:19:21.105967+0000","last_fullsized":"2024-02-14T17:19:21.105967+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-14T17:17:10.961265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-14T17:17:10.961265+0000","last_clean_scrub_stamp":"2024-02-14T17:17:10.961265+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-15T20:35:14.263803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.45300000000000001}]},{"osd":2,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.49199999999999999}]},{"osd":4,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.47499999999999998}]},{"osd":5,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.40400000000000003}]},{"osd":6,"last update":"Wed Feb 14 17:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.439}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.433},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60799999999999998}]},{"osd":3,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55500000000000005}]},{"osd":4,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.629}]},{"osd":5,"last update":"Wed Feb 14 17:19:40 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.67300000000000004}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.68600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:54 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.33700000000000002,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.36499999999999999,"15min":0.33900000000000002},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.57499999999999996,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.46400000000000002}]},{"osd":2,"last update":"Wed Feb 14 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.311,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56799999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.42199999999999999,"15min":0.41299999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.54400000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.309}]},{"osd":3,"last update":"Wed Feb 14 17:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.35099999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.60699999999999998,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.443,"15min":0.44900000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.54000000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.28899999999999998}]},{"osd":4,"last update":"Wed Feb 14 17:20:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.57999999999999996,"15min":0.57199999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.55400000000000005}]},{"osd":5,"last update":"Wed Feb 14 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.625,"15min":0.621},"min":{"1min":0.40000000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.83899999999999997,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.56399999999999995}]},{"osd":6,"last update":"Wed Feb 14 17:19:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.497},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54300000000000004}]},{"osd":7,"last update":"Wed Feb 14 17:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.52300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Feb 14 17:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.35299999999999998,"15min":0.33400000000000002},"min":{"1min":0.28399999999999997,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.55300000000000005,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.36599999999999999,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.50600000000000001,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Wed Feb 14 17:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.40999999999999998,"15min":0.41099999999999998},"min":{"1min":0.25,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56899999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.371},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.47199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.379}]},{"osd":3,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57699999999999996,"5min":0.624,"15min":0.624},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.49199999999999999,"15min":0.49399999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.63900000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.39000000000000001}]},{"osd":4,"last update":"Wed Feb 14 17:19:59 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78000000000000003,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.63900000000000001}]},{"osd":5,"last update":"Wed Feb 14 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58699999999999997,"15min":0.58599999999999997},"min":{"1min":0.41899999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.79600000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63,"15min":0.63300000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77300000000000002,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63300000000000001}]},{"osd":6,"last update":"Wed Feb 14 17:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.61499999999999999}]},{"osd":7,"last update":"Wed Feb 14 17:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.56899999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6264,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738888,"statfs":{"total":95995035648,"available":95988621312,"internally_reserved":0,"allocated":573440,"data_stored":236915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.38100000000000001,"15min":0.38700000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.56899999999999995,"5min":0.628,"15min":0.628},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.42699999999999999}]},{"osd":1,"last update":"Wed Feb 14 17:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39600000000000002,"15min":0.39000000000000001},"min":{"1min":0.27500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60099999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.441},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41099999999999998,"15min":0.39900000000000002},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.63200000000000001,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.41799999999999998}]},{"osd":3,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.53900000000000003,"5min":0.751,"15min":0.751},"last":0.46700000000000003}]},{"osd":4,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.503},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Wed Feb 14 17:20:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59399999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53700000000000003}]},{"osd":6,"last update":"Wed Feb 14 17:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.58399999999999996}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.621}]}]},{"osd":3,"up_from":25,"seq":107374182437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5688,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739464,"statfs":{"total":95995035648,"available":95989211136,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.40500000000000003,"15min":0.40000000000000002},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.438,"15min":0.439},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.58999999999999997,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.29699999999999999}]},{"osd":1,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.439,"15min":0.439},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.436,"15min":0.434},"min":{"1min":0.32200000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65600000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.28399999999999997}]},{"osd":2,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.47499999999999998,"15min":0.46899999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.32800000000000001}]},{"osd":4,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57699999999999996,"15min":0.58199999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77200000000000002,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61799999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.877,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.627,"15min":0.63},"min":{"1min":0.47499999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":0.751,"15min":0.65000000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":14.73,"5min":14.73,"15min":14.73},"last":0.64600000000000002}]},{"osd":6,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.72499999999999998}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.754},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.68100000000000005}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52000000000000002,"15min":0.51800000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72999999999999998,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.53500000000000003}]},{"osd":1,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.50900000000000001,"15min":0.50700000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69399999999999995,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.502},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.42699999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.85099999999999998,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.59299999999999997}]},{"osd":2,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.55100000000000005,"15min":0.55300000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.83299999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70999999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.51700000000000002}]},{"osd":3,"last update":"Wed Feb 14 17:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.33900000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76500000000000001,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53200000000000003,"15min":0.52800000000000002},"min":{"1min":0.36399999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70299999999999996,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.629}]},{"osd":5,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.38800000000000001,"15min":0.377},"min":{"1min":0.28499999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.29399999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45500000000000002}]},{"osd":6,"last update":"Wed Feb 14 17:19:38 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.442},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.434}]},{"osd":7,"last update":"Wed Feb 14 17:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39900000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5688,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739464,"statfs":{"total":95995035648,"available":95989211136,"internally_reserved":0,"allocated":573440,"data_stored":237311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53300000000000003,"15min":0.53700000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.72299999999999998,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.51500000000000001}]},{"osd":1,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52700000000000002,"15min":0.53300000000000003},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54100000000000004}]},{"osd":2,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53700000000000003,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66300000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.81200000000000006,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.629}]},{"osd":3,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55500000000000005,"15min":0.56100000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.64700000000000002,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.50900000000000001}]},{"osd":4,"last update":"Wed Feb 14 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.40699999999999997,"15min":0.40000000000000002},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.65600000000000003,"5min":0.87,"15min":0.87},"last":0.375},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.434,"15min":0.42699999999999999},"min":{"1min":0.32500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63700000000000001,"5min":1.085,"15min":1.085},"last":0.39700000000000002}]},{"osd":6,"last update":"Wed Feb 14 17:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.40899999999999997}]},{"osd":7,"last update":"Wed Feb 14 17:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-14T17:20:29.129 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-14T17:20:29.129 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-14T17:20:29.129 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-14T17:20:29.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- ceph health --format=json 2024-02-14T17:20:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:28 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:20:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:28 smithi111 ceph-mon[25603]: Deploying daemon grafana.a on smithi111 2024-02-14T17:20:29.347 INFO:teuthology.orchestra.run.smithi017.stderr:Inferring config /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/mon.a/config 2024-02-14T17:20:30.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:29 smithi017 ceph-mon[24769]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:20:30.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:29 smithi017 ceph-mon[24769]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:29 smithi017 ceph-mon[29506]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:20:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:29 smithi017 ceph-mon[29506]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:30.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:29 smithi111 ceph-mon[25603]: from='client.14598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-14T17:20:30.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:29 smithi111 ceph-mon[25603]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:30.925 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:30.569Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004212324s 2024-02-14T17:20:31.405 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:20:31.405 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-14T17:20:32.009 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-14T17:20:32.009 INFO:tasks.cephadm:Setup complete, yielding 2024-02-14T17:20:32.009 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-14T17:20:32.025 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-02-14T17:20:32.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-02-14T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:32 smithi017 ceph-mon[29506]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:32.337 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:32 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3937766662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-14T17:20:32.337 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:32 smithi017 ceph-mon[24769]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:32.337 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:32 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3937766662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-14T17:20:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:32 smithi111 ceph-mon[25603]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:32 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3937766662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-14T17:20:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:34 smithi017 ceph-mon[24769]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:34 smithi017 ceph-mon[29506]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:34.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:20:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:34] "GET /metrics HTTP/1.1" 200 191470 "" "Prometheus/2.33.4" 2024-02-14T17:20:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:34 smithi111 ceph-mon[25603]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:35.124 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:20:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "6f3be362-397a-457b-b714-fc46c5194dd9", 2024-02-14T17:20:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "r", 2024-02-14T17:20:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "current_period": "23d55eee-69fb-4237-987c-674bd7fbd51f", 2024-02-14T17:20:35.124 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1 2024-02-14T17:20:35.124 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:20:35.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:35 smithi017 ceph-mon[24769]: osdmap e60: 8 total, 8 up, 8 in 2024-02-14T17:20:35.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:35 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2526824428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:20:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:35 smithi017 ceph-mon[29506]: osdmap e60: 8 total, 8 up, 8 in 2024-02-14T17:20:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2526824428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:20:35.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:35 smithi111 ceph-mon[25603]: osdmap e60: 8 total, 8 up, 8 in 2024-02-14T17:20:35.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:35 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2526824428' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-14T17:20:35.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-02-14T17:20:36.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:36 smithi017 ceph-mon[24769]: pgmap v41: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:36.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:36 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2526824428' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:20:36.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:36 smithi017 ceph-mon[24769]: osdmap e61: 8 total, 8 up, 8 in 2024-02-14T17:20:36.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:36 smithi017 ceph-mon[29506]: pgmap v41: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:36.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:36 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2526824428' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:20:36.411 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:36 smithi017 ceph-mon[29506]: osdmap e61: 8 total, 8 up, 8 in 2024-02-14T17:20:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:36 smithi111 ceph-mon[25603]: pgmap v41: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:20:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:36 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2526824428' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-14T17:20:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:36 smithi111 ceph-mon[25603]: osdmap e61: 8 total, 8 up, 8 in 2024-02-14T17:20:37.403 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:37 smithi017 ceph-mon[24769]: osdmap e62: 8 total, 8 up, 8 in 2024-02-14T17:20:37.676 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:37 smithi017 ceph-mon[29506]: osdmap e62: 8 total, 8 up, 8 in 2024-02-14T17:20:37.676 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "e1fa5334-29f3-4a85-b89d-bac1a5883ec8", 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "default", 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": "true", 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "", 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [], 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [], 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "", 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "6f3be362-397a-457b-b714-fc46c5194dd9", 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2024-02-14T17:20:37.677 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2024-02-14T17:20:37.678 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:37.678 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:20:37.678 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:37 smithi111 ceph-mon[25603]: osdmap e62: 8 total, 8 up, 8 in 2024-02-14T17:20:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:20:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:20:38.162 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-02-14T17:20:38.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:38 smithi017 ceph-mon[24769]: pgmap v44: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-14T17:20:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:38 smithi017 ceph-mon[29506]: pgmap v44: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-14T17:20:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:38 smithi111 ceph-mon[25603]: pgmap v44: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "af2a1170-b0d9-4bd9-a6d9-a75bb4c954f2", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "z", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "domain_root": "z.rgw.meta:root", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "control_pool": "z.rgw.control", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "gc_pool": "z.rgw.log:gc", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "lc_pool": "z.rgw.log:lc", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "log_pool": "z.rgw.log", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-02-14T17:20:39.827 INFO:teuthology.orchestra.run.smithi017.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "roles_pool": "z.rgw.meta:roles", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "otp_pool": "z.rgw.otp", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "system_key": { 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "access_key": "", 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "secret_key": "" 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_pools": [ 2024-02-14T17:20:39.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "default-placement", 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "val": { 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "index_pool": "z.rgw.buckets.index", 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": { 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD": { 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "data_pool": "z.rgw.buckets.data" 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "index_type": 0 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "6f3be362-397a-457b-b714-fc46c5194dd9", 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout: "notif_pool": "z.rgw.log:notif" 2024-02-14T17:20:39.829 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:20:40.374 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:40 smithi111 ceph-mon[25603]: pgmap v45: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-14T17:20:40.397 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:40 smithi017 ceph-mon[24769]: pgmap v45: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-14T17:20:40.397 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:40 smithi017 ceph-mon[29506]: pgmap v45: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-14T17:20:40.421 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-02-14T17:20:41.890 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:41 smithi111 systemd[1]: Starting Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:20:42.151 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:41 smithi111 podman[41711]: 2024-02-14 17:20:41.790767114 +0000 UTC m=+0.018044801 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-14T17:20:42.151 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:41 smithi111 podman[41711]: 2024-02-14 17:20:41.914762114 +0000 UTC m=+0.142039802 container create 2f503a5ea1563bffbf0a354799d3a509bd661480c0a148139f625e8cef40cf72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, io.openshift.expose-services=, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.tags=base rhel8, version=8.5, architecture=x86_64, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182) 2024-02-14T17:20:42.151 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 podman[41711]: 2024-02-14 17:20:42.150607147 +0000 UTC m=+0.377884831 container init 2f503a5ea1563bffbf0a354799d3a509bd661480c0a148139f625e8cef40cf72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, version=8.5, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762) 2024-02-14T17:20:42.401 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-mon[25603]: pgmap v46: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-02-14T17:20:42.401 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 podman[41711]: 2024-02-14 17:20:42.153298866 +0000 UTC m=+0.380576558 container start 2f503a5ea1563bffbf0a354799d3a509bd661480c0a148139f625e8cef40cf72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, release=236.1648460182, name=ubi8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-02-14T17:20:42.401 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-14T17:20:42.401 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-14T17:20:42.401 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-14T17:20:42.402 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="App mode production" logger=settings 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-02-14T17:20:42.403 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-14T17:20:42.404 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-02-14T17:20:42.404 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-02-14T17:20:42.404 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-02-14T17:20:42.404 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-02-14T17:20:42.404 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-02-14T17:20:42.404 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-02-14T17:20:42.405 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-02-14T17:20:42.406 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-02-14T17:20:42.407 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 bash[41711]: 2f503a5ea1563bffbf0a354799d3a509bd661480c0a148139f625e8cef40cf72 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 systemd[1]: Started Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-02-14T17:20:42.408 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-02-14T17:20:42.409 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-02-14T17:20:42.409 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-02-14T17:20:42.409 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-02-14T17:20:42.410 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-02-14T17:20:42.411 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-02-14T17:20:42.412 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-02-14T17:20:42.413 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-14T17:20:42.414 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-02-14T17:20:42.415 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-14T17:20:42.416 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-02-14T17:20:42.417 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-14T17:20:42.417 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-02-14T17:20:42.417 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-02-14T17:20:42.417 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-02-14T17:20:42.417 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-02-14T17:20:42.417 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-02-14T17:20:42.417 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-02-14T17:20:42.418 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-02-14T17:20:42.419 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-02-14T17:20:42.420 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-02-14T17:20:42.420 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-02-14T17:20:42.420 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-02-14T17:20:42.420 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-02-14T17:20:42.423 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-14T17:20:42.423 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-14T17:20:42.423 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-14T17:20:42.423 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-02-14T17:20:42.423 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-02-14T17:20:42.424 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-02-14T17:20:42.425 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-02-14T17:20:42.426 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-02-14T17:20:42.426 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-02-14T17:20:42.426 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-02-14T17:20:42.668 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-02-14T17:20:42.669 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-02-14T17:20:42.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-02-14T17:20:42.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-02-14T17:20:42.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-02-14T17:20:42.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-02-14T17:20:42.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-02-14T17:20:42.671 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-02-14T17:20:42.672 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-02-14T17:20:42.673 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-14T17:20:42.674 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-02-14T17:20:42.675 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-02-14T17:20:42.676 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-02-14T17:20:42.677 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-14T17:20:42.677 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-02-14T17:20:42.677 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-02-14T17:20:42.677 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-02-14T17:20:42.677 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-02-14T17:20:42.677 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-02-14T17:20:42.677 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-02-14T17:20:42.678 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-02-14T17:20:42.679 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=439.079508ms 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-14T17:20:42.680 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-14T17:20:42.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:42 smithi017 ceph-mon[24769]: pgmap v46: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-02-14T17:20:42.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:42 smithi017 ceph-mon[29506]: pgmap v46: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-02-14T17:20:42.974 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-02-14T17:20:42.974 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-14T17:20:42.974 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-14T17:20:42.974 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-14T17:20:42.974 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:20:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:20:42+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: osdmap e63: 8 total, 8 up, 8 in 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1476086614' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:43.646 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:43 smithi111 ceph-mon[25603]: pgmap v48: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: osdmap e63: 8 total, 8 up, 8 in 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1476086614' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[24769]: pgmap v48: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-14T17:20:43.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:43.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:43.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: osdmap e63: 8 total, 8 up, 8 in 2024-02-14T17:20:43.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1476086614' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:20:43.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-02-14T17:20:43.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:43.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:43.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:43 smithi017 ceph-mon[29506]: pgmap v48: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-14T17:20:44.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:44 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-14T17:20:44.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:44 smithi017 ceph-mon[24769]: osdmap e64: 8 total, 8 up, 8 in 2024-02-14T17:20:44.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:20:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:44] "GET /metrics HTTP/1.1" 200 192539 "" "Prometheus/2.33.4" 2024-02-14T17:20:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:44 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-14T17:20:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:44 smithi017 ceph-mon[29506]: osdmap e64: 8 total, 8 up, 8 in 2024-02-14T17:20:44.586 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:44 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-02-14T17:20:44.586 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:44 smithi111 ceph-mon[25603]: osdmap e64: 8 total, 8 up, 8 in 2024-02-14T17:20:45.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[24769]: osdmap e65: 8 total, 8 up, 8 in 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1476086614' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[24769]: pgmap v51: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[29506]: osdmap e65: 8 total, 8 up, 8 in 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1476086614' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:20:45.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:45.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:45 smithi017 ceph-mon[29506]: pgmap v51: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-02-14T17:20:45.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:45 smithi111 ceph-mon[25603]: osdmap e65: 8 total, 8 up, 8 in 2024-02-14T17:20:45.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:45 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1476086614' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:20:45.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:45 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-02-14T17:20:45.696 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:45 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:45.699 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:45 smithi111 ceph-mon[25603]: pgmap v51: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-02-14T17:20:46.407 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[24769]: osdmap e66: 8 total, 8 up, 8 in 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[24769]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[24769]: Reconfiguring daemon alertmanager.a on smithi017 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[29506]: osdmap e66: 8 total, 8 up, 8 in 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.408 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[29506]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:20:46.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[29506]: Reconfiguring daemon alertmanager.a on smithi017 2024-02-14T17:20:46.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:46 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:46 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-02-14T17:20:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:46 smithi111 ceph-mon[25603]: osdmap e66: 8 total, 8 up, 8 in 2024-02-14T17:20:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:46 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:46 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:46 smithi111 ceph-mon[25603]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:20:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:46 smithi111 ceph-mon[25603]: Reconfiguring daemon alertmanager.a on smithi017 2024-02-14T17:20:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:46 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:47.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:47 smithi017 ceph-mon[24769]: osdmap e67: 8 total, 8 up, 8 in 2024-02-14T17:20:47.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:47 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:20:47.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:47 smithi017 ceph-mon[24769]: pgmap v54: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-14T17:20:47.607 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:47 smithi017 ceph-mon[29506]: osdmap e67: 8 total, 8 up, 8 in 2024-02-14T17:20:47.607 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:47 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:20:47.607 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:47 smithi017 ceph-mon[29506]: pgmap v54: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-14T17:20:47.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:20:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:20:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:47 smithi111 ceph-mon[25603]: osdmap e67: 8 total, 8 up, 8 in 2024-02-14T17:20:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:47 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-02-14T17:20:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:47 smithi111 ceph-mon[25603]: pgmap v54: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-14T17:20:48.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:48 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:20:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:48 smithi017 ceph-mon[24769]: osdmap e68: 8 total, 8 up, 8 in 2024-02-14T17:20:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:48 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:20:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:48 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:20:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:48 smithi017 ceph-mon[29506]: osdmap e68: 8 total, 8 up, 8 in 2024-02-14T17:20:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:48 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:20:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:48 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-02-14T17:20:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:48 smithi111 ceph-mon[25603]: osdmap e68: 8 total, 8 up, 8 in 2024-02-14T17:20:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:48 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-14T17:20:49.125 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:48 smithi017 systemd[1]: Stopping Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:20:49.125 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:48 smithi017 bash[49360]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-14T17:20:49.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:20:49.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[24769]: osdmap e69: 8 total, 8 up, 8 in 2024-02-14T17:20:49.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-14T17:20:49.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[24769]: pgmap v57: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-14T17:20:49.401 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:20:49.402 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[29506]: osdmap e69: 8 total, 8 up, 8 in 2024-02-14T17:20:49.402 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-14T17:20:49.402 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-mon[29506]: pgmap v57: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-14T17:20:49.402 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[47225]: level=info ts=2024-02-14T17:20:49.124Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:20:49.402 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:49 smithi017 podman[49373]: 2024-02-14 17:20:49.165100495 +0000 UTC m=+0.167313808 container died 11fd55bbf57f7a9db34a6a20e47617906028407ed1be0633ab0fa5c050a28831 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:49.402 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:20:49.402 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "5920a69c-7525-4587-b69b-f080d6ef1eb3", 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 1, 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "predecessor_uuid": "23d55eee-69fb-4237-987c-674bd7fbd51f", 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_status": [], 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "period_map": { 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "5920a69c-7525-4587-b69b-f080d6ef1eb3", 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "zonegroups": [ 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "e1fa5334-29f3-4a85-b89d-bac1a5883ec8", 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default", 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "api_name": "default", 2024-02-14T17:20:49.403 INFO:teuthology.orchestra.run.smithi017.stdout: "is_master": "true", 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames": [], 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "hostnames_s3website": [], 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "af2a1170-b0d9-4bd9-a6d9-a75bb4c954f2", 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "zones": [ 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "id": "af2a1170-b0d9-4bd9-a6d9-a75bb4c954f2", 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "z", 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "endpoints": [], 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "log_meta": "false", 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "log_data": "false", 2024-02-14T17:20:49.404 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_index_max_shards": 11, 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: "read_only": "false", 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_type": "", 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from_all": "true", 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_from": [], 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: "redirect_zone": "" 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: "placement_targets": [ 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-02-14T17:20:49.405 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "default-placement", 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "tags": [], 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "storage_classes": [ 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "STANDARD" 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "default_placement": "default-placement", 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "6f3be362-397a-457b-b714-fc46c5194dd9", 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "sync_policy": { 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "groups": [] 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: "short_zone_ids": [ 2024-02-14T17:20:49.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "af2a1170-b0d9-4bd9-a6d9-a75bb4c954f2", 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: "val": 598502764 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zonegroup": "e1fa5334-29f3-4a85-b89d-bac1a5883ec8", 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: "master_zone": "af2a1170-b0d9-4bd9-a6d9-a75bb4c954f2", 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: "period_config": { 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_quota": { 2024-02-14T17:20:49.407 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "user_quota": { 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "check_on_raw": false, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size": -1, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "max_size_kb": 0, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "max_objects": -1 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "user_ratelimit": { 2024-02-14T17:20:49.408 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "bucket_ratelimit": { 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "anonymous_ratelimit": { 2024-02-14T17:20:49.409 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_ops": 0, 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_ops": 0, 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: "max_read_bytes": 0, 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: "max_write_bytes": 0, 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: "enabled": false 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_id": "6f3be362-397a-457b-b714-fc46c5194dd9", 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_name": "r", 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout: "realm_epoch": 2 2024-02-14T17:20:49.410 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:20:49.689 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:49 smithi017 podman[49373]: 2024-02-14 17:20:49.499134404 +0000 UTC m=+0.501347716 container cleanup 11fd55bbf57f7a9db34a6a20e47617906028407ed1be0633ab0fa5c050a28831 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:49.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:49 smithi017 bash[49373]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a 2024-02-14T17:20:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:49 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-14T17:20:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:49 smithi111 ceph-mon[25603]: osdmap e69: 8 total, 8 up, 8 in 2024-02-14T17:20:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:49 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-02-14T17:20:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:49 smithi111 ceph-mon[25603]: pgmap v57: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-02-14T17:20:50.076 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-02-14T17:20:50.591 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:50 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-14T17:20:50.591 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:50 smithi017 ceph-mon[29506]: osdmap e70: 8 total, 8 up, 8 in 2024-02-14T17:20:50.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:50 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-14T17:20:50.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:50 smithi017 ceph-mon[24769]: osdmap e70: 8 total, 8 up, 8 in 2024-02-14T17:20:50.592 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:50 smithi017 podman[49387]: 2024-02-14 17:20:50.322006978 +0000 UTC m=+1.156357791 container remove 11fd55bbf57f7a9db34a6a20e47617906028407ed1be0633ab0fa5c050a28831 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:50.592 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:50 smithi017 podman[49387]: 2024-02-14 17:20:50.414323055 +0000 UTC m=+1.248673823 volume remove 47f06caf10e53e46cc46ce9f9d2e518b7e026b7675e40b75ffe01c1bcee2eb2f 2024-02-14T17:20:50.592 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:50 smithi017 bash[49422]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-14T17:20:50.592 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:50 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-14T17:20:50.592 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:50 smithi017 systemd[1]: Stopped Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:50.592 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:50 smithi017 systemd[1]: Starting Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:20:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:50 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3516441904' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-02-14T17:20:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:50 smithi111 ceph-mon[25603]: osdmap e70: 8 total, 8 up, 8 in 2024-02-14T17:20:51.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:51 smithi017 ceph-mon[24769]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 217 B/s rd, 435 B/s wr, 1 op/s 2024-02-14T17:20:51.631 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:51 smithi017 ceph-mon[29506]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 217 B/s rd, 435 B/s wr, 1 op/s 2024-02-14T17:20:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:51 smithi111 ceph-mon[25603]: pgmap v59: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 217 B/s rd, 435 B/s wr, 1 op/s 2024-02-14T17:20:52.162 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled rgw.foo update... 2024-02-14T17:20:52.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:52 smithi017 podman[49578]: 2024-02-14 17:20:51.988304672 +0000 UTC m=+0.448334406 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-14T17:20:52.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:52 smithi017 podman[49578]: 2024-02-14 17:20:52.099701304 +0000 UTC m=+0.559731019 volume create e43d957cb29e51507787da42b22ca64d17e26a45eda6b4dc21f2da3ee9db288a 2024-02-14T17:20:52.439 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:52 smithi017 podman[49578]: 2024-02-14 17:20:52.175013217 +0000 UTC m=+0.635042941 container create b80475d39fdec8e4253b591f918790d7b12efd8e22f70a9226a445410841a12e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:52.838 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph osd pool create foo' 2024-02-14T17:20:53.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-mon[24769]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:20:53.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-mon[24769]: Saving service rgw.foo spec with placement count:2 2024-02-14T17:20:53.433 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:53.434 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-mon[29506]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:20:53.434 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-mon[29506]: Saving service rgw.foo spec with placement count:2 2024-02-14T17:20:53.434 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:53.434 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 podman[49578]: 2024-02-14 17:20:53.223450754 +0000 UTC m=+1.683480470 container init b80475d39fdec8e4253b591f918790d7b12efd8e22f70a9226a445410841a12e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:53.434 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 podman[49578]: 2024-02-14 17:20:53.226140199 +0000 UTC m=+1.686169918 container start b80475d39fdec8e4253b591f918790d7b12efd8e22f70a9226a445410841a12e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:53.434 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.237Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-14T17:20:53.434 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.237Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-14T17:20:53.434 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.238Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-02-14T17:20:53.435 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.239Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-14T17:20:53.435 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.264Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:20:53.435 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.264Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:20:53.435 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.267Z caller=main.go:518 msg=Listening address=:9093 2024-02-14T17:20:53.435 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:53.267Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-14T17:20:53.435 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 bash[49578]: b80475d39fdec8e4253b591f918790d7b12efd8e22f70a9226a445410841a12e 2024-02-14T17:20:53.435 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:53 smithi017 systemd[1]: Started Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:53.441 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:53 smithi111 ceph-mon[25603]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:20:53.441 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:53 smithi111 ceph-mon[25603]: Saving service rgw.foo spec with placement count:2 2024-02-14T17:20:53.441 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:53 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:54 smithi111 ceph-mon[25603]: pgmap v60: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-14T17:20:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:54 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:54 smithi111 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:20:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:54 smithi111 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:20:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[24769]: pgmap v60: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-14T17:20:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[24769]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:20:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[24769]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:20:54.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:20:54] "GET /metrics HTTP/1.1" 200 196806 "" "Prometheus/2.33.4" 2024-02-14T17:20:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[29506]: pgmap v60: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-02-14T17:20:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[29506]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:20:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:54 smithi017 ceph-mon[29506]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:20:55.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:20:55 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:20:55.240Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000967725s 2024-02-14T17:20:56.028 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'foo' created 2024-02-14T17:20:56.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-mon[25603]: pgmap v61: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 931 B/s wr, 2 op/s 2024-02-14T17:20:56.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3136524881' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-14T17:20:56.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:56 smithi017 ceph-mon[24769]: pgmap v61: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 931 B/s wr, 2 op/s 2024-02-14T17:20:56.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:56 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3136524881' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-14T17:20:56.328 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:56 smithi017 ceph-mon[29506]: pgmap v61: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 931 B/s wr, 2 op/s 2024-02-14T17:20:56.329 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:56 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3136524881' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-02-14T17:20:56.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 systemd[1]: Stopping Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:20:56.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 bash[42400]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:20:56.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:20:56.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.328Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.329Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.329Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[40739]: ts=2024-02-14T17:20:56.329Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-14T17:20:56.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 podman[42413]: 2024-02-14 17:20:56.338504757 +0000 UTC m=+0.135727006 container died 91b366745ac6fe83d44ac072578c64f8adff443679125e0f08523d3e9989b1b8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:56.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'rbd pool init foo' 2024-02-14T17:20:57.019 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 podman[42413]: 2024-02-14 17:20:56.761501222 +0000 UTC m=+0.558723477 container cleanup 91b366745ac6fe83d44ac072578c64f8adff443679125e0f08523d3e9989b1b8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:57.019 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 bash[42413]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:20:57.019 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 bash[42440]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:20:57.020 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 podman[42427]: 2024-02-14 17:20:56.970854529 +0000 UTC m=+0.630294562 container remove 91b366745ac6fe83d44ac072578c64f8adff443679125e0f08523d3e9989b1b8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:57.020 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-14T17:20:57.020 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 systemd[1]: Stopped Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:57.020 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:56 smithi111 systemd[1]: Starting Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:20:57.316 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:57 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3136524881' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-14T17:20:57.316 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:57 smithi017 ceph-mon[24769]: osdmap e71: 8 total, 8 up, 8 in 2024-02-14T17:20:57.316 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:57 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3136524881' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-14T17:20:57.316 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:57 smithi017 ceph-mon[29506]: osdmap e71: 8 total, 8 up, 8 in 2024-02-14T17:20:57.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3136524881' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-02-14T17:20:57.339 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-mon[25603]: osdmap e71: 8 total, 8 up, 8 in 2024-02-14T17:20:57.339 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 podman[42506]: 2024-02-14 17:20:57.12130157 +0000 UTC m=+0.018556745 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-14T17:20:57.339 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 podman[42506]: 2024-02-14 17:20:57.222000062 +0000 UTC m=+0.119255265 container create f64ae5e500fe7de359b0f1a4591855660914759ea304b93831780c7aa13dfaab (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:57.685 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 podman[42506]: 2024-02-14 17:20:57.424616182 +0000 UTC m=+0.321871354 container init f64ae5e500fe7de359b0f1a4591855660914759ea304b93831780c7aa13dfaab (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:57.685 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 podman[42506]: 2024-02-14 17:20:57.427473093 +0000 UTC m=+0.324728263 container start f64ae5e500fe7de359b0f1a4591855660914759ea304b93831780c7aa13dfaab (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:20:57.685 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.449Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.449Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.449Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.450Z caller=main.go:518 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi111 (none))" 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.450Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.450Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.451Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.452Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.453Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.455Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.455Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.292µs 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.455Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 bash[42506]: f64ae5e500fe7de359b0f1a4591855660914759ea304b93831780c7aa13dfaab 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 systemd[1]: Started Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:20:57.686 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.666Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-14T17:20:57.687 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.666Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-14T17:20:57.687 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.666Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.016µs wal_replay_duration=210.987962ms total_replay_duration=211.047531ms 2024-02-14T17:20:57.687 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.669Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:20:57.687 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.669Z caller=main.go:947 level=info msg="TSDB started" 2024-02-14T17:20:57.687 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.669Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:20:57.687 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.684Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.377001ms db_storage=849ns remote_storage=1.441µs web_handler=343ns query_engine=587ns scrape=713.416µs scrape_sd=25.668µs notify=18.735µs notify_sd=5.812µs rules=13.327311ms 2024-02-14T17:20:57.687 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:20:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:20:57.684Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-14T17:20:58.323 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: osdmap e72: 8 total, 8 up, 8 in 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: pgmap v64: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 932 B/s wr, 2 op/s 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:58.324 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:58.325 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:58 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: osdmap e72: 8 total, 8 up, 8 in 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: pgmap v64: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 932 B/s wr, 2 op/s 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:20:58.340 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: osdmap e72: 8 total, 8 up, 8 in 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: pgmap v64: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 932 B/s wr, 2 op/s 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.341 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:20:58.342 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:58 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: osdmap e73: 8 total, 8 up, 8 in 2024-02-14T17:20:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1773839345' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: osdmap e73: 8 total, 8 up, 8 in 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1773839345' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-14T17:20:59.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:20:59 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.17:9093"}]: dispatch 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: osdmap e73: 8 total, 8 up, 8 in 2024-02-14T17:20:59.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1773839345' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-14T17:20:59.425 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:20:59 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-02-14T17:21:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:00 smithi017 ceph-mon[24769]: pgmap v66: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:00 smithi017 ceph-mon[24769]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-14T17:21:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:00 smithi017 ceph-mon[24769]: osdmap e74: 8 total, 8 up, 8 in 2024-02-14T17:21:00.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:00 smithi017 ceph-mon[29506]: pgmap v66: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:00.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:00 smithi017 ceph-mon[29506]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-14T17:21:00.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:00 smithi017 ceph-mon[29506]: osdmap e74: 8 total, 8 up, 8 in 2024-02-14T17:21:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:00 smithi111 ceph-mon[25603]: pgmap v66: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:00 smithi111 ceph-mon[25603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-02-14T17:21:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:00 smithi111 ceph-mon[25603]: osdmap e74: 8 total, 8 up, 8 in 2024-02-14T17:21:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: osdmap e75: 8 total, 8 up, 8 in 2024-02-14T17:21:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:21:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:01 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:01.353 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: osdmap e75: 8 total, 8 up, 8 in 2024-02-14T17:21:01.353 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.353 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.353 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: osdmap e75: 8 total, 8 up, 8 in 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.354 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:01.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:01.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:21:01.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:01.355 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:01 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:01.777 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch apply iscsi foo u p' 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[24769]: Saving service rgw.foo spec with placement count:2 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[24769]: Deploying daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[24769]: pgmap v69: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[24769]: osdmap e76: 8 total, 8 up, 8 in 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[29506]: Saving service rgw.foo spec with placement count:2 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[29506]: Deploying daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[29506]: pgmap v69: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:02.333 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:02 smithi017 ceph-mon[29506]: osdmap e76: 8 total, 8 up, 8 in 2024-02-14T17:21:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:02 smithi111 ceph-mon[25603]: Saving service rgw.foo spec with placement count:2 2024-02-14T17:21:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:02 smithi111 ceph-mon[25603]: Deploying daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:21:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:02 smithi111 ceph-mon[25603]: pgmap v69: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:02 smithi111 ceph-mon[25603]: osdmap e76: 8 total, 8 up, 8 in 2024-02-14T17:21:03.365 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:21:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:21:03.242Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002772175s 2024-02-14T17:21:04.366 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:04 smithi017 ceph-mon[24769]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 409 B/s wr, 0 op/s 2024-02-14T17:21:04.366 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:21:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:04] "GET /metrics HTTP/1.1" 200 196806 "" "Prometheus/2.33.4" 2024-02-14T17:21:04.367 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:04 smithi017 ceph-mon[29506]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 409 B/s wr, 0 op/s 2024-02-14T17:21:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:04 smithi111 ceph-mon[25603]: pgmap v71: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 409 B/s wr, 0 op/s 2024-02-14T17:21:04.988 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled iscsi.foo update... 2024-02-14T17:21:05.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'sleep 180' 2024-02-14T17:21:06.274 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: Saving service iscsi.foo spec with placement count:1 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: osdmap e77: 8 total, 8 up, 8 in 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:06.275 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[24769]: Deploying daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:21:06.276 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:21:06.276 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: Saving service iscsi.foo spec with placement count:1 2024-02-14T17:21:06.276 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.276 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.276 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:06.276 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:06.276 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:21:06.277 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: osdmap e77: 8 total, 8 up, 8 in 2024-02-14T17:21:06.277 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.277 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-14T17:21:06.277 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:06.277 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:05 smithi017 ceph-mon[29506]: Deploying daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:21:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:21:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: Saving service iscsi.foo spec with placement count:1 2024-02-14T17:21:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:21:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-14T17:21:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: osdmap e77: 8 total, 8 up, 8 in 2024-02-14T17:21:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-14T17:21:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:05 smithi111 ceph-mon[25603]: Deploying daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:21:07.944 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:21:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:21:08.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:08 smithi111 ceph-mon[25603]: pgmap v74: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 5.0 KiB/s wr, 160 op/s 2024-02-14T17:21:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:08 smithi017 ceph-mon[24769]: pgmap v74: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 5.0 KiB/s wr, 160 op/s 2024-02-14T17:21:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:08 smithi017 ceph-mon[29506]: pgmap v74: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 5.0 KiB/s wr, 160 op/s 2024-02-14T17:21:10.393 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:10 smithi111 ceph-mon[25603]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-14T17:21:10.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:10 smithi017 ceph-mon[24769]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-14T17:21:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:10 smithi017 ceph-mon[29506]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-14T17:21:12.385 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:12 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:12.385 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:12 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:12.385 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:12 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:12.385 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:12 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:12.385 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:12 smithi111 ceph-mon[25603]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-14T17:21:12.385 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:12 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:12.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:12.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:12.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:12.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:12.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[24769]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-14T17:21:12.411 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:12.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:12.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:12.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:12.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:12.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[29506]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-02-14T17:21:12.412 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:12 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:14.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:14 smithi017 ceph-mon[24769]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.4 KiB/s wr, 212 op/s 2024-02-14T17:21:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:14 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:14 smithi017 ceph-mon[29506]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.4 KiB/s wr, 212 op/s 2024-02-14T17:21:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:14 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:21:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:14] "GET /metrics HTTP/1.1" 200 203664 "" "Prometheus/2.33.4" 2024-02-14T17:21:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:14 smithi111 ceph-mon[25603]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.4 KiB/s wr, 212 op/s 2024-02-14T17:21:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:14 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: Checking dashboard <-> RGW credentials 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: Checking pool "foo" exists for service iscsi.foo 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: Deploying daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[24769]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.4 KiB/s wr, 211 op/s 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: Checking dashboard <-> RGW credentials 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: Checking pool "foo" exists for service iscsi.foo 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-14T17:21:15.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:15.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: Deploying daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:21:15.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:15 smithi017 ceph-mon[29506]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.4 KiB/s wr, 211 op/s 2024-02-14T17:21:15.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: Checking dashboard <-> RGW credentials 2024-02-14T17:21:15.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: Checking pool "foo" exists for service iscsi.foo 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: Deploying daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:21:15.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:15 smithi111 ceph-mon[25603]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.4 KiB/s wr, 211 op/s 2024-02-14T17:21:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:16 smithi017 ceph-mon[29506]: mgrmap e21: y(active, since 115s), standbys: x 2024-02-14T17:21:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:16 smithi017 ceph-mon[24769]: mgrmap e21: y(active, since 115s), standbys: x 2024-02-14T17:21:16.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:16 smithi111 ceph-mon[25603]: mgrmap e21: y(active, since 115s), standbys: x 2024-02-14T17:21:17.650 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:17 smithi111 ceph-mon[25603]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 3.0 KiB/s wr, 180 op/s 2024-02-14T17:21:17.650 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:21:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:21:17.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:17 smithi017 ceph-mon[24769]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 3.0 KiB/s wr, 180 op/s 2024-02-14T17:21:17.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:17 smithi017 ceph-mon[29506]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 3.0 KiB/s wr, 180 op/s 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4086174171' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2946607200' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]: dispatch 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]: dispatch 2024-02-14T17:21:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4086174171' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:21:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2946607200' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]: dispatch 2024-02-14T17:21:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:18 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]: dispatch 2024-02-14T17:21:18.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:18 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:18.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:18 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:18.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:18 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:18.908 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:18 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:18.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:18 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4086174171' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:21:18.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:18 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2946607200' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]: dispatch 2024-02-14T17:21:18.909 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:18 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]: dispatch 2024-02-14T17:21:19.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]': finished 2024-02-14T17:21:19.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[24769]: osdmap e78: 8 total, 8 up, 8 in 2024-02-14T17:21:19.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2925589898' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1844808215"}]: dispatch 2024-02-14T17:21:19.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[24769]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 307 B/s wr, 104 op/s 2024-02-14T17:21:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]': finished 2024-02-14T17:21:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[29506]: osdmap e78: 8 total, 8 up, 8 in 2024-02-14T17:21:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2925589898' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1844808215"}]: dispatch 2024-02-14T17:21:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:19 smithi017 ceph-mon[29506]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 307 B/s wr, 104 op/s 2024-02-14T17:21:19.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:19 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2485983711"}]': finished 2024-02-14T17:21:19.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:19 smithi111 ceph-mon[25603]: osdmap e78: 8 total, 8 up, 8 in 2024-02-14T17:21:19.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:19 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2925589898' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1844808215"}]: dispatch 2024-02-14T17:21:19.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:19 smithi111 ceph-mon[25603]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 307 B/s wr, 104 op/s 2024-02-14T17:21:20.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2925589898' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1844808215"}]': finished 2024-02-14T17:21:20.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[24769]: osdmap e79: 8 total, 8 up, 8 in 2024-02-14T17:21:20.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/234085697' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]: dispatch 2024-02-14T17:21:20.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]: dispatch 2024-02-14T17:21:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2925589898' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1844808215"}]': finished 2024-02-14T17:21:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[29506]: osdmap e79: 8 total, 8 up, 8 in 2024-02-14T17:21:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/234085697' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]: dispatch 2024-02-14T17:21:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:20 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]: dispatch 2024-02-14T17:21:20.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:20 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2925589898' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1844808215"}]': finished 2024-02-14T17:21:20.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:20 smithi111 ceph-mon[25603]: osdmap e79: 8 total, 8 up, 8 in 2024-02-14T17:21:20.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:20 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/234085697' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]: dispatch 2024-02-14T17:21:20.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:20 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]: dispatch 2024-02-14T17:21:21.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]': finished 2024-02-14T17:21:21.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: osdmap e80: 8 total, 8 up, 8 in 2024-02-14T17:21:21.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1525882184' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2689455788"}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-02-14T17:21:21.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: Checking dashboard <-> RGW credentials 2024-02-14T17:21:21.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]': finished 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: osdmap e80: 8 total, 8 up, 8 in 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1525882184' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2689455788"}]: dispatch 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-14T17:21:21.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.944 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: Checking dashboard <-> RGW credentials 2024-02-14T17:21:21.945 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2684050458"}]': finished 2024-02-14T17:21:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: osdmap e80: 8 total, 8 up, 8 in 2024-02-14T17:21:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1525882184' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2689455788"}]: dispatch 2024-02-14T17:21:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 341 B/s wr, 7 op/s 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: Adding iSCSI gateway http://:@172.21.15.17:5000 to Dashboard 2024-02-14T17:21:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi017"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:21:21.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: Checking dashboard <-> RGW credentials 2024-02-14T17:21:21.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:21:22.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1525882184' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2689455788"}]': finished 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[24769]: osdmap e81: 8 total, 8 up, 8 in 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3674076412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1844808215"}]: dispatch 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1525882184' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2689455788"}]': finished 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-14T17:21:22.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-14T17:21:22.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-14T17:21:22.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-14T17:21:22.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[29506]: osdmap e81: 8 total, 8 up, 8 in 2024-02-14T17:21:22.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:22 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3674076412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1844808215"}]: dispatch 2024-02-14T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:22 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1525882184' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2689455788"}]': finished 2024-02-14T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-02-14T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-02-14T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-02-14T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-02-14T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:22 smithi111 ceph-mon[25603]: osdmap e81: 8 total, 8 up, 8 in 2024-02-14T17:21:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:22 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3674076412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1844808215"}]: dispatch 2024-02-14T17:21:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:23 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3674076412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1844808215"}]': finished 2024-02-14T17:21:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:23 smithi111 ceph-mon[25603]: osdmap e82: 8 total, 8 up, 8 in 2024-02-14T17:21:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:23 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/509061406' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1905974843"}]: dispatch 2024-02-14T17:21:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:23 smithi111 ceph-mon[25603]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-14T17:21:24.186 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3674076412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1844808215"}]': finished 2024-02-14T17:21:24.186 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[29506]: osdmap e82: 8 total, 8 up, 8 in 2024-02-14T17:21:24.186 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/509061406' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1905974843"}]: dispatch 2024-02-14T17:21:24.186 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[29506]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-14T17:21:24.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3674076412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1844808215"}]': finished 2024-02-14T17:21:24.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[24769]: osdmap e82: 8 total, 8 up, 8 in 2024-02-14T17:21:24.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/509061406' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1905974843"}]: dispatch 2024-02-14T17:21:24.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:23 smithi017 ceph-mon[24769]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-14T17:21:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:21:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:24] "GET /metrics HTTP/1.1" 200 208043 "" "Prometheus/2.33.4" 2024-02-14T17:21:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:24 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/509061406' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1905974843"}]': finished 2024-02-14T17:21:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:24 smithi111 ceph-mon[25603]: osdmap e83: 8 total, 8 up, 8 in 2024-02-14T17:21:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:24 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2787287525' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/996094750"}]: dispatch 2024-02-14T17:21:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:24 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/509061406' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1905974843"}]': finished 2024-02-14T17:21:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:24 smithi017 ceph-mon[24769]: osdmap e83: 8 total, 8 up, 8 in 2024-02-14T17:21:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:24 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2787287525' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/996094750"}]: dispatch 2024-02-14T17:21:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:24 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/509061406' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1905974843"}]': finished 2024-02-14T17:21:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:24 smithi017 ceph-mon[29506]: osdmap e83: 8 total, 8 up, 8 in 2024-02-14T17:21:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:24 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2787287525' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/996094750"}]: dispatch 2024-02-14T17:21:25.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:25 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2787287525' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/996094750"}]': finished 2024-02-14T17:21:25.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:25 smithi111 ceph-mon[25603]: osdmap e84: 8 total, 8 up, 8 in 2024-02-14T17:21:25.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:25 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/969521169' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]: dispatch 2024-02-14T17:21:25.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:25 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]: dispatch 2024-02-14T17:21:25.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:25 smithi111 ceph-mon[25603]: pgmap v90: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-14T17:21:26.044 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2787287525' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/996094750"}]': finished 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[29506]: osdmap e84: 8 total, 8 up, 8 in 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/969521169' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]: dispatch 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]: dispatch 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[29506]: pgmap v90: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2787287525' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/996094750"}]': finished 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[24769]: osdmap e84: 8 total, 8 up, 8 in 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/969521169' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]: dispatch 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]: dispatch 2024-02-14T17:21:26.045 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:25 smithi017 ceph-mon[24769]: pgmap v90: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-02-14T17:21:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]': finished 2024-02-14T17:21:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[24769]: osdmap e85: 8 total, 8 up, 8 in 2024-02-14T17:21:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3337713934' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]: dispatch 2024-02-14T17:21:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]: dispatch 2024-02-14T17:21:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]': finished 2024-02-14T17:21:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[29506]: osdmap e85: 8 total, 8 up, 8 in 2024-02-14T17:21:27.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3337713934' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]: dispatch 2024-02-14T17:21:27.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:27 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]: dispatch 2024-02-14T17:21:27.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:27 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1219630449"}]': finished 2024-02-14T17:21:27.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:27 smithi111 ceph-mon[25603]: osdmap e85: 8 total, 8 up, 8 in 2024-02-14T17:21:27.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:27 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3337713934' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]: dispatch 2024-02-14T17:21:27.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:27 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]: dispatch 2024-02-14T17:21:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:21:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:21:28.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]': finished 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[24769]: osdmap e86: 8 total, 8 up, 8 in 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[24769]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 207 B/s, 0 objects/s recovering 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2260056072' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]: dispatch 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]: dispatch 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]': finished 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[29506]: osdmap e86: 8 total, 8 up, 8 in 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[29506]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 207 B/s, 0 objects/s recovering 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2260056072' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]: dispatch 2024-02-14T17:21:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:28 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]: dispatch 2024-02-14T17:21:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:28 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1994288429"}]': finished 2024-02-14T17:21:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:28 smithi111 ceph-mon[25603]: osdmap e86: 8 total, 8 up, 8 in 2024-02-14T17:21:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:28 smithi111 ceph-mon[25603]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 207 B/s, 0 objects/s recovering 2024-02-14T17:21:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:28 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2260056072' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]: dispatch 2024-02-14T17:21:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:28 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]: dispatch 2024-02-14T17:21:29.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]': finished 2024-02-14T17:21:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[29506]: osdmap e87: 8 total, 8 up, 8 in 2024-02-14T17:21:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4124221623' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1397676400"}]: dispatch 2024-02-14T17:21:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]': finished 2024-02-14T17:21:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[24769]: osdmap e87: 8 total, 8 up, 8 in 2024-02-14T17:21:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:29 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4124221623' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1397676400"}]: dispatch 2024-02-14T17:21:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:29 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:29 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/316814259"}]': finished 2024-02-14T17:21:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:29 smithi111 ceph-mon[25603]: osdmap e87: 8 total, 8 up, 8 in 2024-02-14T17:21:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:29 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4124221623' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1397676400"}]: dispatch 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[24769]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 0 objects/s recovering 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4124221623' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1397676400"}]': finished 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[24769]: osdmap e88: 8 total, 8 up, 8 in 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/463654864' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]: dispatch 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]: dispatch 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[29506]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 0 objects/s recovering 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4124221623' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1397676400"}]': finished 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[29506]: osdmap e88: 8 total, 8 up, 8 in 2024-02-14T17:21:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/463654864' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]: dispatch 2024-02-14T17:21:30.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:30 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]: dispatch 2024-02-14T17:21:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:30 smithi111 ceph-mon[25603]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 191 B/s, 0 objects/s recovering 2024-02-14T17:21:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4124221623' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1397676400"}]': finished 2024-02-14T17:21:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:30 smithi111 ceph-mon[25603]: osdmap e88: 8 total, 8 up, 8 in 2024-02-14T17:21:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/463654864' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]: dispatch 2024-02-14T17:21:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:30 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]: dispatch 2024-02-14T17:21:31.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:31 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]': finished 2024-02-14T17:21:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:31 smithi017 ceph-mon[24769]: osdmap e89: 8 total, 8 up, 8 in 2024-02-14T17:21:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:31 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/256645111' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1666345519"}]: dispatch 2024-02-14T17:21:31.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:31 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]': finished 2024-02-14T17:21:31.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:31 smithi017 ceph-mon[29506]: osdmap e89: 8 total, 8 up, 8 in 2024-02-14T17:21:31.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:31 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/256645111' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1666345519"}]: dispatch 2024-02-14T17:21:31.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:31 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3513237918"}]': finished 2024-02-14T17:21:31.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:31 smithi111 ceph-mon[25603]: osdmap e89: 8 total, 8 up, 8 in 2024-02-14T17:21:31.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:31 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/256645111' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1666345519"}]: dispatch 2024-02-14T17:21:32.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[29506]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/256645111' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1666345519"}]': finished 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[29506]: osdmap e90: 8 total, 8 up, 8 in 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2258859838' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]: dispatch 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]: dispatch 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[24769]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/256645111' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1666345519"}]': finished 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[24769]: osdmap e90: 8 total, 8 up, 8 in 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2258859838' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]: dispatch 2024-02-14T17:21:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:32 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]: dispatch 2024-02-14T17:21:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:32 smithi111 ceph-mon[25603]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:21:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:32 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/256645111' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1666345519"}]': finished 2024-02-14T17:21:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:32 smithi111 ceph-mon[25603]: osdmap e90: 8 total, 8 up, 8 in 2024-02-14T17:21:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:32 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2258859838' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]: dispatch 2024-02-14T17:21:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:32 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]: dispatch 2024-02-14T17:21:33.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]': finished 2024-02-14T17:21:33.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[24769]: osdmap e91: 8 total, 8 up, 8 in 2024-02-14T17:21:33.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3889627972' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]: dispatch 2024-02-14T17:21:33.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]: dispatch 2024-02-14T17:21:33.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]': finished 2024-02-14T17:21:33.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[29506]: osdmap e91: 8 total, 8 up, 8 in 2024-02-14T17:21:33.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3889627972' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]: dispatch 2024-02-14T17:21:33.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:33 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]: dispatch 2024-02-14T17:21:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:33 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/1397676400"}]': finished 2024-02-14T17:21:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:33 smithi111 ceph-mon[25603]: osdmap e91: 8 total, 8 up, 8 in 2024-02-14T17:21:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:33 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3889627972' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]: dispatch 2024-02-14T17:21:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:33 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]: dispatch 2024-02-14T17:21:34.439 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:34] "GET /metrics HTTP/1.1" 200 208043 "" "Prometheus/2.33.4" 2024-02-14T17:21:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[24769]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]': finished 2024-02-14T17:21:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[24769]: osdmap e92: 8 total, 8 up, 8 in 2024-02-14T17:21:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1996869397' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]: dispatch 2024-02-14T17:21:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]: dispatch 2024-02-14T17:21:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[29506]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]': finished 2024-02-14T17:21:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[29506]: osdmap e92: 8 total, 8 up, 8 in 2024-02-14T17:21:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1996869397' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]: dispatch 2024-02-14T17:21:34.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:34 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]: dispatch 2024-02-14T17:21:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:34 smithi111 ceph-mon[25603]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:34 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/1666345519"}]': finished 2024-02-14T17:21:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:34 smithi111 ceph-mon[25603]: osdmap e92: 8 total, 8 up, 8 in 2024-02-14T17:21:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:34 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1996869397' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]: dispatch 2024-02-14T17:21:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:34 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]: dispatch 2024-02-14T17:21:35.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:35 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]': finished 2024-02-14T17:21:35.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:35 smithi017 ceph-mon[24769]: osdmap e93: 8 total, 8 up, 8 in 2024-02-14T17:21:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:35 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]': finished 2024-02-14T17:21:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:35 smithi017 ceph-mon[29506]: osdmap e93: 8 total, 8 up, 8 in 2024-02-14T17:21:35.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:35 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/808657630"}]': finished 2024-02-14T17:21:35.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:35 smithi111 ceph-mon[25603]: osdmap e93: 8 total, 8 up, 8 in 2024-02-14T17:21:36.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:36 smithi017 ceph-mon[24769]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:36 smithi017 ceph-mon[29506]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:36 smithi111 ceph-mon[25603]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:21:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:21:38.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:38 smithi017 ceph-mon[24769]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:21:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:38 smithi017 ceph-mon[29506]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:21:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:38 smithi111 ceph-mon[25603]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:21:39.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:39 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:39 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:39 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:40.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:40 smithi017 ceph-mon[24769]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-14T17:21:40.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:40 smithi017 ceph-mon[29506]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-14T17:21:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:40 smithi111 ceph-mon[25603]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-14T17:21:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:42 smithi017 ceph-mon[24769]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:21:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:42 smithi017 ceph-mon[29506]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:21:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:42 smithi111 ceph-mon[25603]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:21:44.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:44 smithi017 ceph-mon[24769]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:21:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:44 smithi017 ceph-mon[29506]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:21:44.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:21:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:44] "GET /metrics HTTP/1.1" 200 208084 "" "Prometheus/2.33.4" 2024-02-14T17:21:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:44 smithi111 ceph-mon[25603]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:21:46.286 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:46 smithi111 ceph-mon[25603]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-02-14T17:21:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:46 smithi017 ceph-mon[24769]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-02-14T17:21:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:46 smithi017 ceph-mon[29506]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-02-14T17:21:47.697 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:21:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:21:48.418 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:48 smithi111 ceph-mon[25603]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:48.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:48 smithi017 ceph-mon[24769]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:48 smithi017 ceph-mon[29506]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:49.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:49 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:49 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:49 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:50.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:50 smithi017 ceph-mon[24769]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:50 smithi017 ceph-mon[29506]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:50 smithi111 ceph-mon[25603]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:52.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:52 smithi017 ceph-mon[24769]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:52 smithi017 ceph-mon[29506]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:52 smithi111 ceph-mon[25603]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:54 smithi017 ceph-mon[24769]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:54 smithi017 ceph-mon[29506]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:54.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:21:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:54] "GET /metrics HTTP/1.1" 200 208031 "" "Prometheus/2.33.4" 2024-02-14T17:21:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:54 smithi111 ceph-mon[25603]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:56 smithi017 ceph-mon[24769]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:56 smithi017 ceph-mon[29506]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:56 smithi111 ceph-mon[25603]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:21:57.930 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:21:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:21:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:58 smithi111 ceph-mon[25603]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:58 smithi017 ceph-mon[24769]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:58.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:58 smithi017 ceph-mon[29506]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:21:59.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:21:59 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:21:59 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:21:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:21:59 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:00.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:00 smithi017 ceph-mon[24769]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:00.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:00 smithi017 ceph-mon[29506]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:00.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:00 smithi111 ceph-mon[25603]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:02.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:02 smithi017 ceph-mon[24769]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:02.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:02 smithi017 ceph-mon[29506]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:02 smithi111 ceph-mon[25603]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:04.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:04 smithi017 ceph-mon[24769]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:04.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:04 smithi017 ceph-mon[29506]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:04.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:22:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:04] "GET /metrics HTTP/1.1" 200 208031 "" "Prometheus/2.33.4" 2024-02-14T17:22:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:04 smithi111 ceph-mon[25603]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:06.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:06 smithi017 ceph-mon[24769]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:06 smithi017 ceph-mon[29506]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:06 smithi111 ceph-mon[25603]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:22:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:22:08.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:08 smithi017 ceph-mon[24769]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:08 smithi017 ceph-mon[29506]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:08 smithi111 ceph-mon[25603]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:09.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:09 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:09 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:09 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:10.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:10 smithi017 ceph-mon[24769]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:10 smithi017 ceph-mon[29506]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:10 smithi111 ceph-mon[25603]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:12 smithi017 ceph-mon[24769]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:12.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:12 smithi017 ceph-mon[29506]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:12 smithi111 ceph-mon[25603]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:14.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:14 smithi017 ceph-mon[29506]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:14 smithi017 ceph-mon[24769]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:22:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:14] "GET /metrics HTTP/1.1" 200 207998 "" "Prometheus/2.33.4" 2024-02-14T17:22:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:14 smithi111 ceph-mon[25603]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:16.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:16 smithi017 ceph-mon[24769]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:16 smithi017 ceph-mon[29506]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:16 smithi111 ceph-mon[25603]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:22:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:22:18.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:18 smithi017 ceph-mon[24769]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:18 smithi017 ceph-mon[29506]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:18 smithi111 ceph-mon[25603]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:19 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:19.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:19 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:19 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:20 smithi017 ceph-mon[29506]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:20.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:20 smithi017 ceph-mon[24769]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:20 smithi111 ceph-mon[25603]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:21.370 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:22:21.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:22:21.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:22:21.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:22:21.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:22:21.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:22:21.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:22:21.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:22:21.372 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:22:21.372 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:22:21.372 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:22:21.372 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:22:22.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[29506]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[24769]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:22:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:22:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:22 smithi111 ceph-mon[25603]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:22:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:22:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:22:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:22:24.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:24 smithi017 ceph-mon[24769]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:24 smithi017 ceph-mon[29506]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:22:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:24] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-02-14T17:22:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:24 smithi111 ceph-mon[25603]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:26.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:26 smithi017 ceph-mon[24769]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:26 smithi017 ceph-mon[29506]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:26 smithi111 ceph-mon[25603]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:22:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:22:28.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:28 smithi017 ceph-mon[24769]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:28 smithi017 ceph-mon[29506]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:28 smithi111 ceph-mon[25603]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:29.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:29 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:29 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:29 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:30.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:30 smithi017 ceph-mon[24769]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:30 smithi017 ceph-mon[29506]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:30 smithi111 ceph-mon[25603]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:32 smithi017 ceph-mon[24769]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:32 smithi017 ceph-mon[29506]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:32 smithi111 ceph-mon[25603]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:34.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:34 smithi017 ceph-mon[29506]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:34 smithi017 ceph-mon[24769]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:34.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:22:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:34] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-02-14T17:22:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:34 smithi111 ceph-mon[25603]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:36.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:36 smithi017 ceph-mon[24769]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:36 smithi017 ceph-mon[29506]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:36 smithi111 ceph-mon[25603]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:22:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:22:38.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:38 smithi017 ceph-mon[24769]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:38 smithi017 ceph-mon[29506]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:38 smithi111 ceph-mon[25603]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:39.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:39 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:39 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:39 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:40.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:40 smithi017 ceph-mon[24769]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:40.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:40 smithi017 ceph-mon[29506]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:40 smithi111 ceph-mon[25603]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:42.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:42 smithi017 ceph-mon[24769]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:42 smithi017 ceph-mon[29506]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:42 smithi111 ceph-mon[25603]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:44.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:44 smithi017 ceph-mon[24769]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:44 smithi017 ceph-mon[29506]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:44.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:22:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:44] "GET /metrics HTTP/1.1" 200 207992 "" "Prometheus/2.33.4" 2024-02-14T17:22:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:44 smithi111 ceph-mon[25603]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:46.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:46 smithi017 ceph-mon[24769]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:46 smithi017 ceph-mon[29506]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:46 smithi111 ceph-mon[25603]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:22:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:22:48.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:48 smithi017 ceph-mon[24769]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:48 smithi017 ceph-mon[29506]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:48 smithi111 ceph-mon[25603]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:49.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:49 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:49 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:49 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:50.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:50 smithi017 ceph-mon[24769]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:50 smithi017 ceph-mon[29506]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:50 smithi111 ceph-mon[25603]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:52.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:52 smithi017 ceph-mon[24769]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:52 smithi017 ceph-mon[29506]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:52 smithi111 ceph-mon[25603]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:54.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:54 smithi017 ceph-mon[24769]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:54 smithi017 ceph-mon[29506]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:54.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:22:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:54] "GET /metrics HTTP/1.1" 200 207996 "" "Prometheus/2.33.4" 2024-02-14T17:22:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:54 smithi111 ceph-mon[25603]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:56 smithi017 ceph-mon[24769]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:56 smithi017 ceph-mon[29506]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:56 smithi111 ceph-mon[25603]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:22:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:22:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:22:58.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:58 smithi017 ceph-mon[24769]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:58.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:58 smithi017 ceph-mon[29506]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:58 smithi111 ceph-mon[25603]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:22:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:22:59 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:22:59 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:22:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:22:59 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:00.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:00 smithi017 ceph-mon[24769]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:00.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:00 smithi017 ceph-mon[29506]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:00.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:00 smithi111 ceph-mon[25603]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:02.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:02 smithi017 ceph-mon[29506]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:02.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:02 smithi017 ceph-mon[24769]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:02 smithi111 ceph-mon[25603]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:04.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:04 smithi017 ceph-mon[24769]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:04.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:04 smithi017 ceph-mon[29506]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:04.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:23:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:04] "GET /metrics HTTP/1.1" 200 207996 "" "Prometheus/2.33.4" 2024-02-14T17:23:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:04 smithi111 ceph-mon[25603]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:06.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:06 smithi017 ceph-mon[24769]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:06 smithi017 ceph-mon[29506]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:06 smithi111 ceph-mon[25603]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:23:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:23:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:08 smithi017 ceph-mon[24769]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:08 smithi017 ceph-mon[29506]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:08 smithi111 ceph-mon[25603]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:09.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:09 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:09 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:09 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:10.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:10 smithi017 ceph-mon[24769]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:10 smithi017 ceph-mon[29506]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:10 smithi111 ceph-mon[25603]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:12.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:12 smithi017 ceph-mon[29506]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:12 smithi017 ceph-mon[24769]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:12 smithi111 ceph-mon[25603]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:14 smithi017 ceph-mon[24769]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:14 smithi017 ceph-mon[29506]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:23:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:14] "GET /metrics HTTP/1.1" 200 207993 "" "Prometheus/2.33.4" 2024-02-14T17:23:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:14 smithi111 ceph-mon[25603]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:16.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:16 smithi017 ceph-mon[24769]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:16 smithi017 ceph-mon[29506]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:16 smithi111 ceph-mon[25603]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:23:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:23:18.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:18 smithi017 ceph-mon[24769]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:18 smithi017 ceph-mon[29506]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:18 smithi111 ceph-mon[25603]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:19.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:19 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:19.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:19 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:19 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:20.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:20 smithi017 ceph-mon[24769]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:20 smithi017 ceph-mon[29506]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:20 smithi111 ceph-mon[25603]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:23:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:23:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:23:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:23:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:23:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[29506]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[24769]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:23:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:23:22.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:22 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:23:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:22 smithi111 ceph-mon[25603]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:23:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:23:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:23:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:22 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:23:24.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:24 smithi017 ceph-mon[29506]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:24 smithi017 ceph-mon[24769]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:23:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:24] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-02-14T17:23:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:24 smithi111 ceph-mon[25603]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:26 smithi017 ceph-mon[24769]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:26 smithi017 ceph-mon[29506]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:26 smithi111 ceph-mon[25603]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:23:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:23:28.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:28 smithi017 ceph-mon[24769]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:28 smithi017 ceph-mon[29506]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:28 smithi111 ceph-mon[25603]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:29.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:29 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:29 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:29 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:30 smithi017 ceph-mon[24769]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:30 smithi017 ceph-mon[29506]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:30 smithi111 ceph-mon[25603]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:32 smithi017 ceph-mon[24769]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:32 smithi017 ceph-mon[29506]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:32 smithi111 ceph-mon[25603]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:34.425 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:34 smithi017 ceph-mon[24769]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:34.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:23:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:34] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-02-14T17:23:34.426 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:34 smithi017 ceph-mon[29506]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:34 smithi111 ceph-mon[25603]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:36.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:36 smithi017 ceph-mon[24769]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:36 smithi017 ceph-mon[29506]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:36 smithi111 ceph-mon[25603]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:23:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:23:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:38 smithi017 ceph-mon[24769]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:38 smithi017 ceph-mon[29506]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:38 smithi111 ceph-mon[25603]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:39.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:39 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:39 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:39 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:40.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:40 smithi017 ceph-mon[24769]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:40.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:40 smithi017 ceph-mon[29506]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:40 smithi111 ceph-mon[25603]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:42 smithi017 ceph-mon[24769]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:42 smithi017 ceph-mon[29506]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:42 smithi111 ceph-mon[25603]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:44.185 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:44 smithi017 ceph-mon[24769]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:44.185 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:44 smithi017 ceph-mon[29506]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:44.439 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:23:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:44] "GET /metrics HTTP/1.1" 200 207981 "" "Prometheus/2.33.4" 2024-02-14T17:23:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:44 smithi111 ceph-mon[25603]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:46 smithi017 ceph-mon[24769]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:46 smithi017 ceph-mon[29506]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:46 smithi111 ceph-mon[25603]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:23:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:23:48.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:48 smithi017 ceph-mon[24769]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:48 smithi017 ceph-mon[29506]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:48 smithi111 ceph-mon[25603]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:49 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:49 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:49 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:50.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:50 smithi017 ceph-mon[24769]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:50 smithi017 ceph-mon[29506]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:50 smithi111 ceph-mon[25603]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:52.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:52 smithi017 ceph-mon[24769]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:52 smithi017 ceph-mon[29506]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:52 smithi111 ceph-mon[25603]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:54.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:54 smithi017 ceph-mon[24769]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:54 smithi017 ceph-mon[29506]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:54.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:23:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:54] "GET /metrics HTTP/1.1" 200 207936 "" "Prometheus/2.33.4" 2024-02-14T17:23:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:54 smithi111 ceph-mon[25603]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:56.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:56 smithi017 ceph-mon[24769]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:56 smithi017 ceph-mon[29506]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:56 smithi111 ceph-mon[25603]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:23:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:23:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:23:58.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:58 smithi017 ceph-mon[24769]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:58.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:58 smithi017 ceph-mon[29506]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:58 smithi111 ceph-mon[25603]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:23:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:23:59 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:23:59 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:23:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:23:59 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:00.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:00 smithi017 ceph-mon[24769]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:00.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:00 smithi017 ceph-mon[29506]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:00.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:00 smithi111 ceph-mon[25603]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:02.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:02 smithi017 ceph-mon[24769]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:02.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:02 smithi017 ceph-mon[29506]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:02 smithi111 ceph-mon[25603]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:04.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:04 smithi017 ceph-mon[29506]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:04.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:04 smithi017 ceph-mon[24769]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:04.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:24:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:04] "GET /metrics HTTP/1.1" 200 207936 "" "Prometheus/2.33.4" 2024-02-14T17:24:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:04 smithi111 ceph-mon[25603]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:06 smithi017 ceph-mon[24769]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:06 smithi017 ceph-mon[29506]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:06 smithi111 ceph-mon[25603]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:24:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:24:08.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-14T17:24:08.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:08 smithi017 ceph-mon[24769]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:08.336 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:08 smithi017 ceph-mon[29506]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:08 smithi111 ceph-mon[25603]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:09 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:09.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:09 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:09 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:10.183 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:10 smithi017 ceph-mon[24769]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:10.183 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:10 smithi017 ceph-mon[29506]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:10 smithi111 ceph-mon[25603]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:10.499 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-14T17:24:12.435 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:12 smithi017 ceph-mon[24769]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:12.435 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:12 smithi017 ceph-mon[29506]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:12 smithi111 ceph-mon[25603]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:13.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-14T17:24:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:14 smithi017 ceph-mon[24769]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:14 smithi017 ceph-mon[29506]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:24:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:14] "GET /metrics HTTP/1.1" 200 207945 "" "Prometheus/2.33.4" 2024-02-14T17:24:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:14 smithi111 ceph-mon[25603]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:15.390 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:24:16.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:16 smithi017 ceph-mon[24769]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:16.331 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:16 smithi017 ceph-mon[29506]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:16 smithi111 ceph-mon[25603]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:17.256 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:24:17.256 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 2m ago 3m 13.8M - ba2b418f427c b80475d39fde 2024-02-14T17:24:17.256 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (3m) 2m ago 3m 44.3M - 8.3.5 dad864ee21e9 2f503a5ea156 2024-02-14T17:24:17.256 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (3m) 2m ago 2m 68.8M - 3.5 e1d6a67b021e 6f6f2b0cdc81 2024-02-14T17:24:17.257 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e d90044f2e9a4 2024-02-14T17:24:17.257 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (9m) 2m ago 9m 453M - 17.2.0 e1d6a67b021e da74f8a06125 2024-02-14T17:24:17.257 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (9m) 2m ago 10m 46.7M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:24:17.257 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (8m) 2m ago 8m 43.0M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:24:17.257 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (8m) 2m ago 8m 38.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:24:17.257 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 2m ago 4m 16.5M - 1dbe0e931976 dfc53c1158e9 2024-02-14T17:24:17.258 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (4m) 2m ago 4m 18.1M - 1dbe0e931976 8859571f4a9e 2024-02-14T17:24:17.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (7m) 2m ago 7m 47.4M 4017M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:24:17.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (7m) 2m ago 7m 48.2M 4017M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:24:17.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (7m) 2m ago 7m 46.3M 4017M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:24:17.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (6m) 2m ago 6m 43.9M 4017M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:24:17.258 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (6m) 2m ago 6m 45.2M 4272M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:24:17.259 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (6m) 2m ago 6m 46.0M 4272M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:24:17.259 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (5m) 2m ago 5m 42.6M 4272M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:24:17.259 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (5m) 2m ago 5m 47.4M 4272M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:24:17.259 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (3m) 2m ago 4m 43.6M - 514e6a882f6e f64ae5e500fe 2024-02-14T17:24:17.259 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (3m) 2m ago 3m 71.0M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:24:17.259 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (3m) 2m ago 3m 71.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:24:17.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:24:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:24:17.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions' 2024-02-14T17:24:18.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:18 smithi017 ceph-mon[24769]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:18 smithi017 ceph-mon[29506]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:18 smithi111 ceph-mon[25603]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:19.421 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:19 smithi017 ceph-mon[24769]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:19.421 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:19 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:19.421 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:19 smithi017 ceph-mon[29506]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:19.422 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:19 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:19 smithi111 ceph-mon[25603]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:19 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:24:19.906 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:24:19.907 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:24:19.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:24:19.907 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:24:19.907 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-02-14T17:24:19.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:24:19.907 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:24:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:20 smithi017 ceph-mon[24769]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:20 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3422643185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:24:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:20 smithi017 ceph-mon[29506]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:20 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3422643185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:24:20.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph -s' 2024-02-14T17:24:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:20 smithi111 ceph-mon[25603]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:20 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3422643185' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:24:21.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:24:21.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:24:21.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:24:21.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:24:21.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:24:21.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:24:21.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:24:21.410 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:24:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:24:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:24:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:24:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:24:22.374 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:22 smithi111 ceph-mon[25603]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:22.375 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:24:22.375 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:22.375 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:22 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:24:22.375 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[29506]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:22.375 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:24:22.375 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:22.375 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:24:22.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[24769]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:22.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:24:22.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:22.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:22 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:24:22.385 INFO:teuthology.orchestra.run.smithi017.stdout: cluster: 2024-02-14T17:24:22.385 INFO:teuthology.orchestra.run.smithi017.stdout: id: 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: health: HEALTH_OK 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: services: 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: mgr: y(active, since 5m), standbys: x 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: data: 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: pools: 6 pools, 161 pgs 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: objects: 209 objects, 585 KiB 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: pgs: 161 active+clean 2024-02-14T17:24:22.386 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:24:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: io: 2024-02-14T17:24:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: client: 852 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-14T17:24:22.387 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:24:23.033 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ls' 2024-02-14T17:24:23.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:23 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2359012219' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:24:23.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:23 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:23.579 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:23 smithi017 ceph-mon[24769]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:23.579 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:23 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2359012219' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:24:23.579 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:23 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:23.580 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:23 smithi017 ceph-mon[29506]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:23 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2359012219' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:24:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:23 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:23 smithi111 ceph-mon[25603]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:24.379 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:24:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:24] "GET /metrics HTTP/1.1" 200 207944 "" "Prometheus/2.33.4" 2024-02-14T17:24:24.774 INFO:teuthology.orchestra.run.smithi017.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-14T17:24:24.774 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi017=a;count:1 2024-02-14T17:24:24.774 INFO:teuthology.orchestra.run.smithi017.stdout:grafana ?:3000 1/1 3m ago 4m smithi111=a;count:1 2024-02-14T17:24:24.774 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-02-14T17:24:24.775 INFO:teuthology.orchestra.run.smithi017.stdout:mgr 2/2 3m ago 8m smithi017=y;smithi111=x;count:2 2024-02-14T17:24:24.775 INFO:teuthology.orchestra.run.smithi017.stdout:mon 3/3 3m ago 8m smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi111:172.21.15.111=b;count:3 2024-02-14T17:24:24.775 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi017=a;smithi111=b;count:2 2024-02-14T17:24:24.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd 8 3m ago - 2024-02-14T17:24:24.775 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus ?:9095 1/1 3m ago 5m smithi111=a;count:1 2024-02-14T17:24:24.775 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-02-14T17:24:25.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-14T17:24:26.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:26 smithi017 ceph-mon[24769]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:26.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:26 smithi017 ceph-mon[24769]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:26.330 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:26 smithi017 ceph-mon[29506]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:26.330 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:26 smithi017 ceph-mon[29506]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:26 smithi111 ceph-mon[25603]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:26 smithi111 ceph-mon[25603]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:27.566 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled to redeploy mgr.x on host 'smithi111' 2024-02-14T17:24:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:24:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:24:28.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps --refresh' 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[24769]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2621964021' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[29506]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2621964021' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:28 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:24:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:28 smithi111 ceph-mon[25603]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:28 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2621964021' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:24:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:28 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:28 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:24:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:29 smithi017 ceph-mon[24769]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:29 smithi017 ceph-mon[24769]: Schedule redeploy daemon mgr.x 2024-02-14T17:24:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:29 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:29 smithi017 ceph-mon[29506]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:29 smithi017 ceph-mon[29506]: Schedule redeploy daemon mgr.x 2024-02-14T17:24:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:29 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:29 smithi111 ceph-mon[25603]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:29 smithi111 ceph-mon[25603]: Schedule redeploy daemon mgr.x 2024-02-14T17:24:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:29 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:30 smithi017 ceph-mon[24769]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:30 smithi017 ceph-mon[29506]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:30 smithi111 ceph-mon[25603]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 3m ago 4m 13.8M - ba2b418f427c b80475d39fde 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (3m) 3m ago 3m 44.3M - 8.3.5 dad864ee21e9 2f503a5ea156 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (3m) 3m ago 3m 68.8M - 3.5 e1d6a67b021e 6f6f2b0cdc81 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e d90044f2e9a4 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (10m) 3m ago 10m 453M - 17.2.0 e1d6a67b021e da74f8a06125 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (10m) 3m ago 10m 46.7M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (8m) 3m ago 8m 43.0M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (8m) 3m ago 8m 38.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 3m ago 4m 16.5M - 1dbe0e931976 dfc53c1158e9 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (4m) 3m ago 4m 18.1M - 1dbe0e931976 8859571f4a9e 2024-02-14T17:24:30.508 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (8m) 3m ago 8m 47.4M 4017M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (7m) 3m ago 7m 48.2M 4017M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (7m) 3m ago 7m 46.3M 4017M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (7m) 3m ago 7m 43.9M 4017M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (6m) 3m ago 6m 45.2M 4272M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (6m) 3m ago 6m 46.0M 4272M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (5m) 3m ago 5m 42.6M 4272M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (5m) 3m ago 5m 47.4M 4272M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (3m) 3m ago 4m 43.6M - 514e6a882f6e f64ae5e500fe 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (3m) 3m ago 3m 71.0M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:24:30.509 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (3m) 3m ago 3m 71.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:24:31.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'sleep 180' 2024-02-14T17:24:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:32 smithi017 ceph-mon[29506]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:32 smithi017 ceph-mon[29506]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:32 smithi017 ceph-mon[24769]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:32 smithi017 ceph-mon[24769]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:32 smithi111 ceph-mon[25603]: from='client.14988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:24:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:32 smithi111 ceph-mon[25603]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-14T17:24:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:34 smithi017 ceph-mon[24769]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:34 smithi017 ceph-mon[29506]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:34.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:24:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:34] "GET /metrics HTTP/1.1" 200 207944 "" "Prometheus/2.33.4" 2024-02-14T17:24:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:34 smithi111 ceph-mon[25603]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:36 smithi017 ceph-mon[24769]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:36 smithi017 ceph-mon[29506]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:36 smithi111 ceph-mon[25603]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:24:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:24:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:38 smithi017 ceph-mon[24769]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:38 smithi017 ceph-mon[29506]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:38 smithi111 ceph-mon[25603]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:39.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:39 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:39 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:39 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:40.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:40 smithi017 ceph-mon[24769]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:40.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:40 smithi017 ceph-mon[29506]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:40 smithi111 ceph-mon[25603]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:42.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:42 smithi017 ceph-mon[24769]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:42 smithi017 ceph-mon[29506]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:42 smithi111 ceph-mon[25603]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:44.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:44 smithi017 ceph-mon[24769]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:44 smithi017 ceph-mon[29506]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:44.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:24:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:44] "GET /metrics HTTP/1.1" 200 207934 "" "Prometheus/2.33.4" 2024-02-14T17:24:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:44 smithi111 ceph-mon[25603]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:46 smithi017 ceph-mon[24769]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:46 smithi017 ceph-mon[29506]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:46 smithi111 ceph-mon[25603]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:24:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:24:48.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:48 smithi017 ceph-mon[24769]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:48 smithi017 ceph-mon[29506]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:48 smithi111 ceph-mon[25603]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:49.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:49 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:49 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:49 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:50.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:50 smithi017 ceph-mon[24769]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:50 smithi017 ceph-mon[29506]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:50 smithi111 ceph-mon[25603]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:52.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:52 smithi017 ceph-mon[24769]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:52 smithi017 ceph-mon[29506]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:52 smithi111 ceph-mon[25603]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:53.806 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:24:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:24:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:24:53.806 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:24:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:24:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:24:54.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:54 smithi017 ceph-mon[24769]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:54.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:54 smithi017 ceph-mon[29506]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:54.442 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:24:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:54] "GET /metrics HTTP/1.1" 200 207923 "" "Prometheus/2.33.4" 2024-02-14T17:24:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:54 smithi111 ceph-mon[25603]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:55 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:55 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:55 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:24:56.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:55 smithi111 ceph-mon[25603]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:56.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:56.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:56.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:24:56.424 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[24769]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:56.424 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:56.424 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:56.424 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:24:56.425 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:55 smithi017 ceph-mon[29506]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:24:57.815 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:24:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:24:58.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:58 smithi111 ceph-mon[25603]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:58.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:58 smithi017 ceph-mon[24769]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:58.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:58 smithi017 ceph-mon[29506]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:24:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:24:59.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:24:59 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:24:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:24:59 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:25:00.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:00 smithi017 ceph-mon[24769]: Deploying daemon mgr.x on smithi111 2024-02-14T17:25:00.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:00 smithi017 ceph-mon[24769]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:00.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:00 smithi017 ceph-mon[29506]: Deploying daemon mgr.x on smithi111 2024-02-14T17:25:00.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:00 smithi017 ceph-mon[29506]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:00.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:00 smithi111 ceph-mon[25603]: Deploying daemon mgr.x on smithi111 2024-02-14T17:25:00.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:00 smithi111 ceph-mon[25603]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:02.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:02 smithi017 ceph-mon[24769]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:02.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:02 smithi017 ceph-mon[29506]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:02 smithi111 ceph-mon[25603]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:03.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:25:03.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:03.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:03.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:04.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:04 smithi017 ceph-mon[24769]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:04.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:04 smithi017 ceph-mon[29506]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:04.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:25:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:04] "GET /metrics HTTP/1.1" 200 207923 "" "Prometheus/2.33.4" 2024-02-14T17:25:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:04 smithi111 ceph-mon[25603]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:06 smithi017 ceph-mon[24769]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:06 smithi017 ceph-mon[29506]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:06 smithi111 ceph-mon[25603]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:25:08.163 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:08 smithi017 ceph-mon[24769]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:08.439 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:08 smithi017 ceph-mon[29506]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:08 smithi111 ceph-mon[25603]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:09 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:09 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:09 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:10 smithi017 ceph-mon[24769]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:10 smithi017 ceph-mon[29506]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:10 smithi111 ceph-mon[25603]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:12.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:12 smithi017 ceph-mon[24769]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:12.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:12 smithi017 ceph-mon[29506]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:12 smithi111 ceph-mon[25603]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:25:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:25:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:14 smithi017 ceph-mon[24769]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:14 smithi017 ceph-mon[29506]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:25:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:14] "GET /metrics HTTP/1.1" 200 207939 "" "Prometheus/2.33.4" 2024-02-14T17:25:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:14 smithi111 ceph-mon[25603]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:16 smithi017 ceph-mon[24769]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:16 smithi017 ceph-mon[29506]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:16 smithi111 ceph-mon[25603]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[26781]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:25:18.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:18 smithi017 ceph-mon[24769]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:18.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:18 smithi017 ceph-mon[29506]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:18 smithi111 ceph-mon[25603]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:19 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:19.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:19 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:19 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:20.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:20 smithi017 ceph-mon[24769]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:20.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:20 smithi017 ceph-mon[29506]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:20 smithi111 ceph-mon[25603]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:25:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:25:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:25:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:25:21.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:25:21.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:25:21.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:25:21.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:25:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:25:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:25:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:25:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:25:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:22 smithi017 ceph-mon[24769]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:22 smithi017 ceph-mon[29506]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:22 smithi111 ceph-mon[25603]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:25:23.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:24.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:24 smithi017 ceph-mon[24769]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:24 smithi017 ceph-mon[29506]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:25:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:24] "GET /metrics HTTP/1.1" 200 207948 "" "Prometheus/2.33.4" 2024-02-14T17:25:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:24 smithi111 ceph-mon[25603]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:24.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:24 smithi111 systemd[1]: Stopping Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:25:24.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:24 smithi111 bash[45667]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr.x" found: no such container 2024-02-14T17:25:24.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:24 smithi111 podman[45677]: 2024-02-14 17:25:24.749374092 +0000 UTC m=+0.131275844 container died d90044f2e9a43bea02916ba67e058d1aeed159403b26351cb8afb8efa6148149 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, release=754, vcs-type=git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vendor=Red Hat, Inc.) 2024-02-14T17:25:25.414 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 podman[45677]: 2024-02-14 17:25:25.111997586 +0000 UTC m=+0.493899314 container cleanup d90044f2e9a43bea02916ba67e058d1aeed159403b26351cb8afb8efa6148149 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, architecture=x86_64, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.expose-services=, name=centos-stream, release=754, vcs-type=git, version=8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD) 2024-02-14T17:25:25.414 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 bash[45677]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x 2024-02-14T17:25:25.414 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 bash[45707]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr.x" found: no such container 2024-02-14T17:25:25.689 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:25.229Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:25:25.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 podman[45691]: 2024-02-14 17:25:25.413587355 +0000 UTC m=+0.657722538 container remove d90044f2e9a43bea02916ba67e058d1aeed159403b26351cb8afb8efa6148149 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vcs-type=git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, release=754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8) 2024-02-14T17:25:25.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:25:25.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-14T17:25:25.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 systemd[1]: Stopped Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:25:25.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service: Consumed 16.794s CPU time. 2024-02-14T17:25:25.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 systemd[1]: Starting Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:25:26.080 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 podman[45813]: 2024-02-14 17:25:25.814488064 +0000 UTC m=+0.107058339 container create 2ffaf8e8fcebcc1093c727b93a86bc2362f13aa4619222deaca6b3ae20792a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-14T17:25:26.080 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:25 smithi111 podman[45813]: 2024-02-14 17:25:25.726049385 +0000 UTC m=+0.018619669 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:25:26.081 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:26 smithi111 podman[45813]: 2024-02-14 17:25:26.016798426 +0000 UTC m=+0.309368701 container init 2ffaf8e8fcebcc1093c727b93a86bc2362f13aa4619222deaca6b3ae20792a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:25:26.081 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:26 smithi111 podman[45813]: 2024-02-14 17:25:26.019458853 +0000 UTC m=+0.312029129 container start 2ffaf8e8fcebcc1093c727b93a86bc2362f13aa4619222deaca6b3ae20792a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-14T17:25:26.081 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:26 smithi111 ceph-mgr[45832]: -- 172.21.15.111:0/3128086125 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5563b13021a0 con 0x5563b12d9000 2024-02-14T17:25:26.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:26 smithi017 ceph-mon[24769]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:26 smithi017 ceph-mon[29506]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:26 smithi111 ceph-mon[25603]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:26.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:26 smithi111 bash[45813]: 2ffaf8e8fcebcc1093c727b93a86bc2362f13aa4619222deaca6b3ae20792a4b 2024-02-14T17:25:26.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:26 smithi111 systemd[1]: Started Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:25:26.965 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:26.604+0000 7fa5cf07d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:25:26.965 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:26.702+0000 7fa5cf07d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:25:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:25:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:25:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:25:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:25:27.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:25:27.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:25:27.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:27 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:27.596 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:27.387+0000 7fa5cf07d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:25:27.596 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:27.480+0000 7fa5cf07d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:25:27.870 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:27.646+0000 7fa5cf07d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:25:27.870 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:27.744+0000 7fa5cf07d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:25:28.166 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:27.834+0000 7fa5cf07d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:25:28.166 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:28.111+0000 7fa5cf07d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:25:28.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:28 smithi017 ceph-mon[24769]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:28 smithi017 ceph-mon[29506]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:28.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:28 smithi111 ceph-mon[25603]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:28.420 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:28.228+0000 7fa5cf07d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:25:28.420 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:28.320+0000 7fa5cf07d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:25:28.420 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:28.418+0000 7fa5cf07d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:25:28.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:28.599+0000 7fa5cf07d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:25:29.373 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:29 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:29 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:29 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:29.949 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:29.644+0000 7fa5cf07d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:25:29.949 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:29.775+0000 7fa5cf07d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:25:30.213 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:30 smithi111 ceph-mon[25603]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:30.213 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:30.119+0000 7fa5cf07d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:25:30.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:30 smithi017 ceph-mon[24769]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:30 smithi017 ceph-mon[29506]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:30.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:30.212+0000 7fa5cf07d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:25:30.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:30.423+0000 7fa5cf07d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:25:30.912 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:30.618+0000 7fa5cf07d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:25:31.188 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:30.910+0000 7fa5cf07d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:25:31.439 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:31.187+0000 7fa5cf07d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:25:31.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:31.437+0000 7fa5cf07d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:25:32.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:31 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:32.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:31 smithi111 ceph-mon[25603]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:32.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:31 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:32.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:31 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:32.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:31 smithi017 ceph-mon[24769]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:32.242 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:31 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:32.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:31 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:32.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:31 smithi017 ceph-mon[29506]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:32.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:31 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:25:32.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:32.247+0000 7fa5cf07d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:25:33.139 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:32.814+0000 7fa5cf07d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:25:33.139 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:32.915+0000 7fa5cf07d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:25:33.458 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:33.247Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:33.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[24769]: Standby manager daemon x restarted 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[24769]: Standby manager daemon x started 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[29506]: Standby manager daemon x restarted 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:25:33.459 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:25:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-mon[25603]: Standby manager daemon x restarted 2024-02-14T17:25:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-mon[25603]: Standby manager daemon x started 2024-02-14T17:25:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:25:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:25:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:25:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:25:33.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:33.137+0000 7fa5cf07d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:25:33.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:25:33] ENGINE Bus STARTING 2024-02-14T17:25:33.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CherryPy Checker: 2024-02-14T17:25:33.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: The Application mounted at '' has an empty config. 2024-02-14T17:25:33.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:25:33.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:25:33] ENGINE Serving on http://:::9283 2024-02-14T17:25:33.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:25:33] ENGINE Bus STARTED 2024-02-14T17:25:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:25:33.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:34.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:34 smithi017 ceph-mon[24769]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:34 smithi017 ceph-mon[24769]: mgrmap e22: y(active, since 6m), standbys: x 2024-02-14T17:25:34.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:25:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:34] "GET /metrics HTTP/1.1" 200 207948 "" "Prometheus/2.33.4" 2024-02-14T17:25:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:34 smithi017 ceph-mon[29506]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:34 smithi017 ceph-mon[29506]: mgrmap e22: y(active, since 6m), standbys: x 2024-02-14T17:25:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:34 smithi111 ceph-mon[25603]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:34 smithi111 ceph-mon[25603]: mgrmap e22: y(active, since 6m), standbys: x 2024-02-14T17:25:36.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:36 smithi017 ceph-mon[24769]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:36 smithi017 ceph-mon[29506]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:36 smithi111 ceph-mon[25603]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:25:38.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:38 smithi017 ceph-mon[24769]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:38.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:38 smithi017 ceph-mon[29506]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:38 smithi111 ceph-mon[25603]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:39.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:39 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:39 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:39 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:40.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:40 smithi017 ceph-mon[29506]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:40.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:40 smithi017 ceph-mon[24769]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:40 smithi111 ceph-mon[25603]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:42.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:42 smithi017 ceph-mon[24769]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:42.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:42 smithi017 ceph-mon[29506]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:42 smithi111 ceph-mon[25603]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:25:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:44.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:44 smithi017 ceph-mon[24769]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:44 smithi017 ceph-mon[29506]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:44.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:25:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:44] "GET /metrics HTTP/1.1" 200 207955 "" "Prometheus/2.33.4" 2024-02-14T17:25:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:44 smithi111 ceph-mon[25603]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:46 smithi017 ceph-mon[24769]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:46 smithi017 ceph-mon[29506]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:46 smithi111 ceph-mon[25603]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:47.706 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:25:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:48 smithi017 ceph-mon[24769]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:48.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:48 smithi017 ceph-mon[29506]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:48.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:48 smithi111 ceph-mon[25603]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:49.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:49 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:49 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:49 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:50.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:50 smithi017 ceph-mon[24769]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:50 smithi017 ceph-mon[29506]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:50 smithi111 ceph-mon[25603]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:52.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:52 smithi017 ceph-mon[24769]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:52 smithi017 ceph-mon[29506]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:52 smithi111 ceph-mon[25603]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:25:53.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:25:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:25:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:25:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:25:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:54 smithi017 ceph-mon[24769]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:54 smithi017 ceph-mon[29506]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:54.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:25:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:54] "GET /metrics HTTP/1.1" 200 207940 "" "Prometheus/2.33.4" 2024-02-14T17:25:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:54 smithi111 ceph-mon[25603]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:56 smithi017 ceph-mon[24769]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:56 smithi017 ceph-mon[29506]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:56 smithi111 ceph-mon[25603]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:25:57.943 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:25:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:25:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:58 smithi111 ceph-mon[25603]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:58.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:58 smithi017 ceph-mon[24769]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:58.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:58 smithi017 ceph-mon[29506]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:25:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:25:59 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:25:59 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:25:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:25:59 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:00.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:00 smithi017 ceph-mon[24769]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:00.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:00 smithi017 ceph-mon[29506]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:00.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:00 smithi111 ceph-mon[25603]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:02.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:02 smithi017 ceph-mon[24769]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:02.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:02 smithi017 ceph-mon[29506]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:02 smithi111 ceph-mon[25603]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:03.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:26:03.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:03.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:03.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:03.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:04.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:04 smithi017 ceph-mon[24769]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:04.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:04 smithi017 ceph-mon[29506]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:04.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:26:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:04] "GET /metrics HTTP/1.1" 200 207940 "" "Prometheus/2.33.4" 2024-02-14T17:26:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:04 smithi111 ceph-mon[25603]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:06 smithi017 ceph-mon[24769]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:06 smithi017 ceph-mon[29506]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:06 smithi111 ceph-mon[25603]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:26:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:26:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:08 smithi017 ceph-mon[24769]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:08 smithi017 ceph-mon[29506]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:08 smithi111 ceph-mon[25603]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:09 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:09 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:09 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:10 smithi017 ceph-mon[24769]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:10 smithi017 ceph-mon[29506]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:10 smithi111 ceph-mon[25603]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:12 smithi017 ceph-mon[24769]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:12.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:12 smithi017 ceph-mon[29506]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:12 smithi111 ceph-mon[25603]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:26:13.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:13.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:13.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:14.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:14 smithi017 ceph-mon[24769]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:14 smithi017 ceph-mon[29506]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:26:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:14] "GET /metrics HTTP/1.1" 200 207949 "" "Prometheus/2.33.4" 2024-02-14T17:26:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:14 smithi111 ceph-mon[25603]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:16 smithi017 ceph-mon[24769]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:16 smithi017 ceph-mon[29506]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:16 smithi111 ceph-mon[25603]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:17.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:26:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:26:18.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:18 smithi017 ceph-mon[24769]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:18 smithi017 ceph-mon[29506]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:18 smithi111 ceph-mon[25603]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:19 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:19.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:19 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:19 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:20.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:20 smithi017 ceph-mon[24769]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:20 smithi017 ceph-mon[29506]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:20 smithi111 ceph-mon[25603]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:26:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:26:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:26:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:26:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:26:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:22 smithi111 ceph-mon[25603]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:22.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:22 smithi017 ceph-mon[24769]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:22 smithi017 ceph-mon[29506]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:26:23.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:23.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:23.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:24.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:24 smithi017 ceph-mon[24769]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:24 smithi017 ceph-mon[29506]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:26:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:24] "GET /metrics HTTP/1.1" 200 207953 "" "Prometheus/2.33.4" 2024-02-14T17:26:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:24 smithi111 ceph-mon[25603]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:26.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:26 smithi017 ceph-mon[24769]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:26 smithi017 ceph-mon[29506]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:26 smithi111 ceph-mon[25603]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:26:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:26:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:28 smithi017 ceph-mon[24769]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:28 smithi017 ceph-mon[29506]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:28 smithi111 ceph-mon[25603]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:29 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:29 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:29 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:30 smithi111 ceph-mon[25603]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:30 smithi017 ceph-mon[24769]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:30 smithi017 ceph-mon[29506]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:32 smithi111 ceph-mon[25603]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:32 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:26:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:32 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:26:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:32 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:26:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:32 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[24769]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[29506]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:26:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:26:32.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:26:32.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:32 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:26:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:26:33.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:34 smithi017 ceph-mon[24769]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:34 smithi017 ceph-mon[29506]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:34.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:26:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:34] "GET /metrics HTTP/1.1" 200 207953 "" "Prometheus/2.33.4" 2024-02-14T17:26:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:34 smithi111 ceph-mon[25603]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:36 smithi017 ceph-mon[24769]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:36 smithi017 ceph-mon[29506]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:36 smithi111 ceph-mon[25603]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:26:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:26:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:38 smithi017 ceph-mon[24769]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:38 smithi017 ceph-mon[29506]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:38 smithi111 ceph-mon[25603]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:39 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:39.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:39 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:39 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:40 smithi111 ceph-mon[25603]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:40 smithi017 ceph-mon[24769]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:40 smithi017 ceph-mon[29506]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:42 smithi111 ceph-mon[25603]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:42.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:42 smithi017 ceph-mon[24769]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:42 smithi017 ceph-mon[29506]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:26:43.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:43.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:44.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:44 smithi017 ceph-mon[24769]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:44 smithi017 ceph-mon[29506]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:44.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:26:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:44] "GET /metrics HTTP/1.1" 200 207957 "" "Prometheus/2.33.4" 2024-02-14T17:26:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:44 smithi111 ceph-mon[25603]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:46 smithi017 ceph-mon[24769]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:46 smithi017 ceph-mon[29506]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:46 smithi111 ceph-mon[25603]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:26:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:26:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:48 smithi017 ceph-mon[24769]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:48 smithi017 ceph-mon[29506]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:48 smithi111 ceph-mon[25603]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:49 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:49.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:49 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:49.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:49 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:50 smithi111 ceph-mon[25603]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:50 smithi017 ceph-mon[24769]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:50 smithi017 ceph-mon[29506]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:52 smithi111 ceph-mon[25603]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:52 smithi017 ceph-mon[24769]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:52 smithi017 ceph-mon[29506]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:26:53.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:26:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:26:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:26:53.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:26:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:54 smithi017 ceph-mon[24769]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:54 smithi017 ceph-mon[29506]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:54.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:26:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:54] "GET /metrics HTTP/1.1" 200 207951 "" "Prometheus/2.33.4" 2024-02-14T17:26:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:54 smithi111 ceph-mon[25603]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:56 smithi017 ceph-mon[24769]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:56 smithi017 ceph-mon[29506]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:56 smithi111 ceph-mon[25603]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:26:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:26:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:26:58.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:58 smithi017 ceph-mon[24769]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:58.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:58 smithi017 ceph-mon[29506]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:58 smithi111 ceph-mon[25603]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:26:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:26:59 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:59.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:26:59 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:26:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:26:59 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:00.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:00 smithi111 ceph-mon[25603]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:00 smithi017 ceph-mon[24769]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:00 smithi017 ceph-mon[29506]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:02.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:02 smithi111 ceph-mon[25603]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:02.492 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:02 smithi017 ceph-mon[24769]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:02.492 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:02 smithi017 ceph-mon[29506]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:03.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:27:03.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:27:03.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:03.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:27:03.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:04.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:04 smithi017 ceph-mon[24769]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:04.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:04 smithi017 ceph-mon[29506]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:04.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:04] "GET /metrics HTTP/1.1" 200 207951 "" "Prometheus/2.33.4" 2024-02-14T17:27:04.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:04 smithi111 ceph-mon[25603]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:06 smithi017 ceph-mon[24769]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:06 smithi017 ceph-mon[29506]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:06 smithi111 ceph-mon[25603]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:27:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:08 smithi017 ceph-mon[24769]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:08 smithi017 ceph-mon[29506]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:08 smithi111 ceph-mon[25603]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:09 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:09.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:09 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:09 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:10 smithi111 ceph-mon[25603]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:10 smithi017 ceph-mon[24769]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:10 smithi017 ceph-mon[29506]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:12 smithi111 ceph-mon[25603]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:12 smithi017 ceph-mon[24769]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:12 smithi017 ceph-mon[29506]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:27:13.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:13.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:27:13.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:13.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:14 smithi017 ceph-mon[24769]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:14 smithi017 ceph-mon[29506]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:14] "GET /metrics HTTP/1.1" 200 207953 "" "Prometheus/2.33.4" 2024-02-14T17:27:14.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:14 smithi111 ceph-mon[25603]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:16.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:16 smithi017 ceph-mon[24769]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:16 smithi017 ceph-mon[29506]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:16 smithi111 ceph-mon[25603]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:27:18.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:18 smithi017 ceph-mon[24769]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:18 smithi017 ceph-mon[29506]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:18 smithi111 ceph-mon[25603]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:19.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:19 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:19.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:19 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:19 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:20.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:20 smithi111 ceph-mon[25603]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:20.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:20 smithi017 ceph-mon[24769]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:20 smithi017 ceph-mon[29506]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:21 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:21 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:21 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:22 smithi111 ceph-mon[25603]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:22 smithi017 ceph-mon[24769]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:22 smithi017 ceph-mon[29506]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:27:23.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:27:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:24 smithi017 ceph-mon[24769]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:24 smithi017 ceph-mon[29506]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:24] "GET /metrics HTTP/1.1" 200 207948 "" "Prometheus/2.33.4" 2024-02-14T17:27:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:24 smithi111 ceph-mon[25603]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:26.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:26 smithi017 ceph-mon[24769]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:26 smithi017 ceph-mon[29506]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:26 smithi111 ceph-mon[25603]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:27:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:28 smithi017 ceph-mon[24769]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:28 smithi017 ceph-mon[29506]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:28 smithi111 ceph-mon[25603]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:29 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:29 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:29 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:30.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:30 smithi111 ceph-mon[25603]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:30 smithi017 ceph-mon[24769]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:30 smithi017 ceph-mon[29506]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[24769]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:27:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[24769]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:27:32.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[24769]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:27:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[29506]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:32.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:32.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:27:32.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[29506]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:27:32.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:32 smithi017 ceph-mon[29506]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:27:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:32 smithi111 ceph-mon[25603]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:32 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:32 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:27:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:32 smithi111 ceph-mon[25603]: from='mgr.24340 172.21.15.17:0/1200322897' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:27:32.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:32 smithi111 ceph-mon[25603]: from='mgr.24340 ' entity='mgr.y' 2024-02-14T17:27:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:27:33.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:33.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:33.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:27:34.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:34 smithi017 ceph-mon[24769]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:34.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:34 smithi017 ceph-mon[29506]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:34.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:34] "GET /metrics HTTP/1.1" 200 207948 "" "Prometheus/2.33.4" 2024-02-14T17:27:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:34 smithi111 ceph-mon[25603]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:35.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:27:36.246 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:36 smithi017 ceph-mon[24769]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:36.246 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:36 smithi017 ceph-mon[29506]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:36 smithi111 ceph-mon[25603]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:37.834 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (6m) 2m ago 7m 24.4M - ba2b418f427c b80475d39fde 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (6m) 2m ago 6m 54.1M - 8.3.5 dad864ee21e9 2f503a5ea156 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (6m) 2m ago 6m 50.9M - 3.5 e1d6a67b021e 6f6f2b0cdc81 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283 running (2m) 2m ago 11m 67.6M - 19.0.0-1430-g8727ce93 170846cf6432 2ffaf8e8fceb 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (13m) 2m ago 13m 461M - 17.2.0 e1d6a67b021e da74f8a06125 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (13m) 2m ago 13m 47.5M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (11m) 2m ago 11m 43.6M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (12m) 2m ago 12m 39.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (7m) 2m ago 7m 16.8M - 1dbe0e931976 dfc53c1158e9 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (7m) 2m ago 7m 16.7M - 1dbe0e931976 8859571f4a9e 2024-02-14T17:27:37.835 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (11m) 2m ago 11m 50.0M 4017M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (10m) 2m ago 10m 50.0M 4017M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (10m) 2m ago 10m 47.8M 4017M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (10m) 2m ago 10m 47.2M 4017M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (9m) 2m ago 9m 48.9M 4272M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (9m) 2m ago 9m 49.7M 4272M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (9m) 2m ago 9m 45.5M 4272M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (8m) 2m ago 8m 50.8M 4272M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (6m) 2m ago 7m 67.5M - 514e6a882f6e f64ae5e500fe 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (6m) 2m ago 6m 72.0M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:27:37.836 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (6m) 2m ago 6m 72.2M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:27:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:27:38.416 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:38 smithi017 ceph-mon[24769]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:38.416 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:38 smithi017 ceph-mon[29506]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:38.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions' 2024-02-14T17:27:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:38 smithi111 ceph-mon[25603]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:39.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:39 smithi017 ceph-mon[24769]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:27:39.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:39 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:39 smithi017 ceph-mon[29506]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:27:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:39 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:39 smithi111 ceph-mon[25603]: from='client.24916 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:27:39.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:39 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:40.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:40 smithi017 ceph-mon[24769]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:40.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:40 smithi017 ceph-mon[29506]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:40 smithi111 ceph-mon[25603]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 1 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:27:40.510 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 1 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:27:40.511 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:27:41.129 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph -s' 2024-02-14T17:27:41.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:41 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4180636822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:27:41.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:41 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4180636822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:27:41.544 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:41 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4180636822' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:27:42.235 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:42 smithi017 ceph-mon[24769]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:42.235 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:42 smithi017 ceph-mon[29506]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:42.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:42 smithi111 ceph-mon[25603]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: cluster: 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: id: 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: health: HEALTH_OK 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: services: 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: mon: 3 daemons, quorum a,c,b (age 11m) 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: mgr: y(active, since 8m), standbys: x 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 8m) 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-14T17:27:42.951 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: data: 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: pools: 6 pools, 161 pgs 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: objects: 209 objects, 585 KiB 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: pgs: 161 active+clean 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: io: 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-14T17:27:42.952 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:27:43.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:43 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1209283989' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:27:43.523 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:43 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1209283989' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:27:43.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:43 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1209283989' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:27:43.624 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:27:43.932 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:27:43.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:27:43.932 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:43.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:43.932 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:43.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:27:44.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:44 smithi017 ceph-mon[24769]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:44.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:44 smithi017 ceph-mon[29506]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:44.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:44] "GET /metrics HTTP/1.1" 200 207952 "" "Prometheus/2.33.4" 2024-02-14T17:27:44.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:44 smithi111 ceph-mon[25603]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:45.603 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-02-14T17:27:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:46 smithi017 ceph-mon[24769]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:46 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/940056462' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:27:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:46 smithi017 ceph-mon[29506]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:46 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/940056462' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:27:46.221 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-02-14T17:27:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:46 smithi111 ceph-mon[25603]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:46 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/940056462' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:27:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:27:48.292 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:48 smithi017 ceph-mon[24769]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:48.292 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:48 smithi017 ceph-mon[29506]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:48.292 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:27:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:48 smithi111 ceph-mon[25603]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:27:48.889 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph mgr fail' 2024-02-14T17:27:49.429 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:49 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:49.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:49 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2882127344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:27:49.430 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:49 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:49.430 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:49 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2882127344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:27:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:49 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:49.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:49 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2882127344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:27:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:50 smithi111 ceph-mon[25603]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:50.496 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:50 smithi017 ceph-mon[24769]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:50.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:50 smithi017 ceph-mon[29506]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:27:51.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2296277312' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:27:51.479 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-mon[24769]: osdmap e94: 8 total, 8 up, 8 in 2024-02-14T17:27:51.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2296277312' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:27:51.479 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-mon[29506]: osdmap e94: 8 total, 8 up, 8 in 2024-02-14T17:27:51.480 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setuser ceph since I am not root 2024-02-14T17:27:51.480 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ignoring --setgroup ceph since I am not root 2024-02-14T17:27:51.480 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-mgr[25006]: -- 172.21.15.17:0/1491045907 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb1f78c340 con 0x55fb1f765000 2024-02-14T17:27:51.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:27:51] ENGINE Bus STOPPING 2024-02-14T17:27:51.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:27:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:27:51.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:27:51] ENGINE Bus STOPPED 2024-02-14T17:27:51.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:27:51] ENGINE Bus STARTING 2024-02-14T17:27:51.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:27:51] ENGINE Serving on http://:::9283 2024-02-14T17:27:51.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:27:51] ENGINE Bus STARTED 2024-02-14T17:27:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2296277312' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:27:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:51 smithi111 ceph-mon[25603]: osdmap e94: 8 total, 8 up, 8 in 2024-02-14T17:27:52.024 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:51.749+0000 7f28c14cf000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:27:52.024 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:51.844+0000 7f28c14cf000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:27:52.047 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'sleep 180' 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2296277312' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: mgrmap e23: x(active, starting, since 0.169232s) 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:27:52.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: Manager daemon x is now available 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: Queued rgw.foo for migration 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:52.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: Deploying cephadm binary to smithi017 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[24769]: Deploying cephadm binary to smithi111 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2296277312' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:27:52.370 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: mgrmap e23: x(active, starting, since 0.169232s) 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:27:52.371 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: Manager daemon x is now available 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: Queued rgw.foo for migration 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: Deploying cephadm binary to smithi017 2024-02-14T17:27:52.372 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-mon[29506]: Deploying cephadm binary to smithi111 2024-02-14T17:27:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2296277312' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: mgrmap e23: x(active, starting, since 0.169232s) 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:27:52.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: Manager daemon x is now available 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: Queued rgw.foo for migration 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:27:52.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:52.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:27:52.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:52.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:27:52.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: Deploying cephadm binary to smithi017 2024-02-14T17:27:52.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:52 smithi111 ceph-mon[25603]: Deploying cephadm binary to smithi111 2024-02-14T17:27:52.644 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:52.476+0000 7f28c14cf000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:27:52.644 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:52.570+0000 7f28c14cf000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:27:52.898 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:52.660+0000 7f28c14cf000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:27:52.898 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:52.813+0000 7f28c14cf000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:27:52.898 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:52 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:52.896+0000 7f28c14cf000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:27:53.189 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:53.010+0000 7f28c14cf000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:27:53.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:53.099+0000 7f28c14cf000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:27:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:53 smithi111 ceph-mon[25603]: mgrmap e24: x(active, since 1.18348s) 2024-02-14T17:27:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:53 smithi111 ceph-mon[25603]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:53 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:53 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:53 smithi111 ceph-mon[25603]: [14/Feb/2024:17:27:53] ENGINE Bus STARTING 2024-02-14T17:27:53.522 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[24769]: mgrmap e24: x(active, since 1.18348s) 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[24769]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[24769]: [14/Feb/2024:17:27:53] ENGINE Bus STARTING 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[29506]: mgrmap e24: x(active, since 1.18348s) 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[29506]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:27:53.523 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-mon[29506]: [14/Feb/2024:17:27:53] ENGINE Bus STARTING 2024-02-14T17:27:53.524 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:53.267+0000 7f28c14cf000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:27:53.524 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:53.357+0000 7f28c14cf000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:27:53.854 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:27:53.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:53.854 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:27:53.854 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:27:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:27:53.529Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:27:54.435 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[29506]: [14/Feb/2024:17:27:53] ENGINE Serving on http://172.21.15.111:8765 2024-02-14T17:27:54.435 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[29506]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:54.435 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[29506]: [14/Feb/2024:17:27:53] ENGINE Serving on https://172.21.15.111:7150 2024-02-14T17:27:54.435 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[29506]: [14/Feb/2024:17:27:53] ENGINE Bus STARTED 2024-02-14T17:27:54.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[24769]: [14/Feb/2024:17:27:53] ENGINE Serving on http://172.21.15.111:8765 2024-02-14T17:27:54.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[24769]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:54.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[24769]: [14/Feb/2024:17:27:53] ENGINE Serving on https://172.21.15.111:7150 2024-02-14T17:27:54.436 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-mon[24769]: [14/Feb/2024:17:27:53] ENGINE Bus STARTED 2024-02-14T17:27:54.689 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:54.433+0000 7f28c14cf000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:27:54.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:54.566+0000 7f28c14cf000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:27:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:54 smithi111 ceph-mon[25603]: [14/Feb/2024:17:27:53] ENGINE Serving on http://172.21.15.111:8765 2024-02-14T17:27:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:54 smithi111 ceph-mon[25603]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:54 smithi111 ceph-mon[25603]: [14/Feb/2024:17:27:53] ENGINE Serving on https://172.21.15.111:7150 2024-02-14T17:27:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:54 smithi111 ceph-mon[25603]: [14/Feb/2024:17:27:53] ENGINE Bus STARTED 2024-02-14T17:27:55.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:54.763+0000 7f28c14cf000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:27:55.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:54.854+0000 7f28c14cf000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:27:55.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:54.995+0000 7f28c14cf000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:27:55.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:55 smithi017 ceph-mon[24769]: mgrmap e25: x(active, since 3s) 2024-02-14T17:27:55.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:55 smithi017 ceph-mon[29506]: mgrmap e25: x(active, since 3s) 2024-02-14T17:27:55.555 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:55 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:55.323+0000 7f28c14cf000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:27:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:55 smithi111 ceph-mon[25603]: mgrmap e25: x(active, since 3s) 2024-02-14T17:27:55.819 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:55 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:55.553+0000 7f28c14cf000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:27:55.819 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:55 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:55.641+0000 7f28c14cf000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:27:55.819 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:55 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:55.817+0000 7f28c14cf000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:27:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:56 smithi017 ceph-mon[24769]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:56 smithi017 ceph-mon[29506]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:56 smithi111 ceph-mon[25603]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:57.114 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:56 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:56.778+0000 7f28c14cf000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:27:57.114 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:57.112+0000 7f28c14cf000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:27:57.384 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:57.211+0000 7f28c14cf000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:27:57.384 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:57.382+0000 7f28c14cf000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:27:57.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:27:57] ENGINE Bus STARTING 2024-02-14T17:27:57.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: CherryPy Checker: 2024-02-14T17:27:57.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: The Application mounted at '' has an empty config. 2024-02-14T17:27:57.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: 2024-02-14T17:27:57.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:27:57] ENGINE Serving on http://:::9283 2024-02-14T17:27:57.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:27:57 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: [14/Feb/2024:17:27:57] ENGINE Bus STARTED 2024-02-14T17:27:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:27:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:27:57] "GET /metrics HTTP/1.1" 200 34889 "" "Prometheus/2.33.4" 2024-02-14T17:27:58.349 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[29506]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:58.349 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[29506]: Standby manager daemon y started 2024-02-14T17:27:58.349 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[24769]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[24769]: Standby manager daemon y started 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:27:58.350 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:58 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:58 smithi111 ceph-mon[25603]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:27:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:58 smithi111 ceph-mon[25603]: Standby manager daemon y started 2024-02-14T17:27:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:58 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:27:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:58 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:27:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:58 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:27:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:58 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/832893028' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:27:58.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:58 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:27:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:59 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:27:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:27:59 smithi017 ceph-mon[24769]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-14T17:27:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:59 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:27:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:27:59 smithi017 ceph-mon[29506]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-14T17:27:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:59 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:27:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:27:59 smithi111 ceph-mon[25603]: mgrmap e26: x(active, since 7s), standbys: y 2024-02-14T17:28:00.189 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:28:00.032Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:28:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:00 smithi017 ceph-mon[29506]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:28:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:00 smithi017 ceph-mon[24769]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:28:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:00 smithi111 ceph-mon[25603]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:01 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:01 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:01 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:01 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:01 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:28:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi017 to 2993M 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: Adjusting osd_memory_target on smithi111 to 3248M 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:28:02.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi017 to 2993M 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: Adjusting osd_memory_target on smithi111 to 3248M 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:28:02.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:28:02.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:28:02.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:28:02.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:28:02.698 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:02 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:28:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:28:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi017 to 2993M 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: Adjusting osd_memory_target on smithi111 to 3248M 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:28:02.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:02 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:28:03.690 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:03 smithi017 systemd[1]: Stopping Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:03.690 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:03 smithi017 bash[58665]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter.a" found: no such container 2024-02-14T17:28:03.690 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:03 smithi017 podman[58679]: 2024-02-14 17:28:03.555384774 +0000 UTC m=+0.112455267 container died dfc53c1158e99a6eb829a1f3159da33140058102d8058c98dce862d782d0d869 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:03.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=error ts=2024-02-14T17:28:03.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:28:03.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:28:03.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.17:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.17 because it doesn't contain any IP SANs" 2024-02-14T17:28:03.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=warn ts=2024-02-14T17:28:03.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.111:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.111 because it doesn't contain any IP SANs" 2024-02-14T17:28:04.186 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:03 smithi017 podman[58679]: 2024-02-14 17:28:03.936260331 +0000 UTC m=+0.493330826 container cleanup dfc53c1158e99a6eb829a1f3159da33140058102d8058c98dce862d782d0d869 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:04.186 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:03 smithi017 bash[58679]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a 2024-02-14T17:28:04.186 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:03 smithi017 bash[58705]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter.a" found: no such container 2024-02-14T17:28:04.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:28:04.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:28:04.186 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[24769]: Reconfiguring daemon node-exporter.a on smithi017 2024-02-14T17:28:04.187 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-14T17:28:04.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:03 smithi017 ceph-mon[29506]: Reconfiguring daemon node-exporter.a on smithi017 2024-02-14T17:28:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:28:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:28:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:28:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:28:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: Reconfiguring node-exporter.a (dependencies changed)... 2024-02-14T17:28:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:03 smithi111 ceph-mon[25603]: Reconfiguring daemon node-exporter.a on smithi017 2024-02-14T17:28:04.439 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 podman[58693]: 2024-02-14 17:28:04.204636823 +0000 UTC m=+0.653130187 container remove dfc53c1158e99a6eb829a1f3159da33140058102d8058c98dce862d782d0d869 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 systemd[1]: Stopped Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Consumed 3.673s CPU time. 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 systemd[1]: Starting Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 podman[58774]: 2024-02-14 17:28:04.421578295 +0000 UTC m=+0.112889138 container create f9e29bdc31dee85ef65f67417f71eadc13b79b51eaab900c8db4c7feda948bc3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:04.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 podman[58774]: 2024-02-14 17:28:04.326633115 +0000 UTC m=+0.017943965 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-14T17:28:04.906 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 podman[58774]: 2024-02-14 17:28:04.623622119 +0000 UTC m=+0.314932961 container init f9e29bdc31dee85ef65f67417f71eadc13b79b51eaab900c8db4c7feda948bc3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:04.906 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 podman[58774]: 2024-02-14 17:28:04.626338606 +0000 UTC m=+0.317649453 container start f9e29bdc31dee85ef65f67417f71eadc13b79b51eaab900c8db4c7feda948bc3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:04.906 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.629Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-14T17:28:04.906 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.629Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-14T17:28:04.906 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=arp 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-14T17:28:04.907 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=edac 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-14T17:28:04.908 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.630Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=os 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-14T17:28:04.909 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=stat 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=time 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=uname 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[58789]: ts=2024-02-14T17:28:04.631Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 bash[58774]: f9e29bdc31dee85ef65f67417f71eadc13b79b51eaab900c8db4c7feda948bc3 2024-02-14T17:28:04.910 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:28:04 smithi017 systemd[1]: Started Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-mon[29506]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:28:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:05.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-mon[24769]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:28:05.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:05.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:04 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:04 smithi111 ceph-mon[25603]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:28:05.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:04 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:05.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:04 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:06.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:05 smithi017 ceph-mon[24769]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:28:06.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:05 smithi017 ceph-mon[24769]: Reconfiguring daemon alertmanager.a on smithi017 2024-02-14T17:28:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:05 smithi017 ceph-mon[29506]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:28:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:05 smithi017 ceph-mon[29506]: Reconfiguring daemon alertmanager.a on smithi017 2024-02-14T17:28:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:05 smithi111 ceph-mon[25603]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-14T17:28:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:05 smithi111 ceph-mon[25603]: Reconfiguring daemon alertmanager.a on smithi017 2024-02-14T17:28:07.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:06 smithi017 ceph-mon[24769]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:06 smithi017 ceph-mon[29506]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:06 smithi111 ceph-mon[25603]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:07] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-02-14T17:28:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:08 smithi017 ceph-mon[24769]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:08 smithi017 ceph-mon[24769]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:09.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:08 smithi017 ceph-mon[29506]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:09.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:08 smithi017 ceph-mon[29506]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:09.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:08 smithi111 ceph-mon[25603]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:09.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:08 smithi111 ceph-mon[25603]: from='client.24712 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:11.128 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:10 smithi017 ceph-mon[24769]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:11.129 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:10 smithi017 ceph-mon[29506]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:10 smithi111 ceph-mon[25603]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:28:12.190 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 systemd[1]: Stopping Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:12.190 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 bash[59040]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-14T17:28:12.487 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[49670]: level=info ts=2024-02-14T17:28:12.193Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:28:12.487 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 podman[59051]: 2024-02-14 17:28:12.214296114 +0000 UTC m=+0.122765864 container died b80475d39fdec8e4253b591f918790d7b12efd8e22f70a9226a445410841a12e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:12.734 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:12.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.111:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:28:12.836 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 podman[59051]: 2024-02-14 17:28:12.601706302 +0000 UTC m=+0.510176043 container cleanup b80475d39fdec8e4253b591f918790d7b12efd8e22f70a9226a445410841a12e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:12.837 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 bash[59051]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a 2024-02-14T17:28:12.837 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 bash[59078]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager.a" found: no such container 2024-02-14T17:28:13.173 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:12 smithi017 ceph-mon[24769]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:13.173 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:12 smithi017 ceph-mon[29506]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:13.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 podman[59065]: 2024-02-14 17:28:12.836025915 +0000 UTC m=+0.618206057 container remove b80475d39fdec8e4253b591f918790d7b12efd8e22f70a9226a445410841a12e (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:13.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 podman[59065]: 2024-02-14 17:28:12.936811056 +0000 UTC m=+0.718991201 volume remove e43d957cb29e51507787da42b22ca64d17e26a45eda6b4dc21f2da3ee9db288a 2024-02-14T17:28:13.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-14T17:28:13.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 systemd[1]: Stopped Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:13.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service: Consumed 1.075s CPU time. 2024-02-14T17:28:13.174 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:12 smithi017 systemd[1]: Starting Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:12 smithi111 ceph-mon[25603]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:13.440 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 podman[59151]: 2024-02-14 17:28:13.075723861 +0000 UTC m=+0.018499173 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-02-14T17:28:13.440 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 podman[59151]: 2024-02-14 17:28:13.239390168 +0000 UTC m=+0.182165469 volume create e46bf647d1f4cadffc7bca027e6cc57fc01bf773aa2596e37be38401dd08966d 2024-02-14T17:28:13.440 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 podman[59151]: 2024-02-14 17:28:13.339891222 +0000 UTC m=+0.282666517 container create 556f84c8c0ade76aaa9fbd0293dfce888636e013f6ce9476edef8c9ed0da0aff (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 podman[59151]: 2024-02-14 17:28:13.642806477 +0000 UTC m=+0.585581773 container init 556f84c8c0ade76aaa9fbd0293dfce888636e013f6ce9476edef8c9ed0da0aff (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 podman[59151]: 2024-02-14 17:28:13.645399301 +0000 UTC m=+0.588174598 container start 556f84c8c0ade76aaa9fbd0293dfce888636e013f6ce9476edef8c9ed0da0aff (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.655Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.655Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.656Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.657Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.683Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.683Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.685Z caller=main.go:518 msg=Listening address=:9093 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:13.685Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 bash[59151]: 556f84c8c0ade76aaa9fbd0293dfce888636e013f6ce9476edef8c9ed0da0aff 2024-02-14T17:28:13.867 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:13 smithi017 systemd[1]: Started Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:14.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:28:15.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:28:15.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:14 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:28:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:28:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:28:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:28:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:28:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:14 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:28:15.936 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:15 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:15.658Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000998368s 2024-02-14T17:28:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:16 smithi017 ceph-mon[24769]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:16.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:16 smithi017 ceph-mon[29506]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:16.382 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-mon[25603]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:17.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:17 smithi111 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.235 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.236 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:28:17.237 INFO:journalctl@ceph.mon.b.smithi111.stdout: return future.result(timeout) 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.__get_result() 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: raise self._exception 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.238 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:17 smithi111 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:28:16.900+0000 7fa554d34700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:28:17.241 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return future.result(timeout) 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.__get_result() 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: raise self._exception 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: config_blobs=daemon_spec.final_config, 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:28:17.242 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.243 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.244 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.244 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.244 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.244 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.244 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.244 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:16 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:17 smithi017 ceph-mon[24769]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:28:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.441 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:28:17.442 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.443 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:17 smithi017 ceph-mon[24769]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:17 smithi017 ceph-mon[29506]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.444 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.445 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:28:17.446 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.447 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:28:17.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:17 smithi017 ceph-mon[29506]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:28:17.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:17] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-02-14T17:28:18.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:18 smithi017 ceph-mon[24769]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:28:18.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:18 smithi017 ceph-mon[24769]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:18.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:18 smithi017 ceph-mon[29506]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:28:18.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:18 smithi017 ceph-mon[29506]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:18 smithi111 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:28:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:18 smithi111 ceph-mon[25603]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:20 smithi017 ceph-mon[24769]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:20 smithi017 ceph-mon[29506]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:20 smithi111 ceph-mon[25603]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:22 smithi017 ceph-mon[24769]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:22 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:22 smithi017 ceph-mon[29506]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:22 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:22 smithi111 ceph-mon[25603]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:22 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:22.734 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.17:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.111:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:28:23.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:28:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:28:23.661Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003395448s 2024-02-14T17:28:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:24 smithi017 ceph-mon[24769]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:24 smithi017 ceph-mon[29506]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:24.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:28:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[25002]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:28:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:24 smithi111 ceph-mon[25603]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:26 smithi017 ceph-mon[24769]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:26 smithi017 ceph-mon[29506]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:26.450 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:26 smithi111 ceph-mon[25603]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:27.884 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:27] "GET /metrics HTTP/1.1" 200 37890 "" "Prometheus/2.33.4" 2024-02-14T17:28:28.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:27 smithi111 systemd[1]: Stopping Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:28.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 bash[49033]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:28:28.524 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-mon[25603]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.239Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:28:28.525 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:28:28.526 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-14T17:28:28.526 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[42520]: ts=2024-02-14T17:28:28.240Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-14T17:28:28.526 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 podman[49044]: 2024-02-14 17:28:28.250078444 +0000 UTC m=+0.217566263 container died f64ae5e500fe7de359b0f1a4591855660914759ea304b93831780c7aa13dfaab (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:28.540 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:28 smithi017 ceph-mon[24769]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:28.540 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:28 smithi017 ceph-mon[29506]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:28.875 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 podman[49044]: 2024-02-14 17:28:28.631568441 +0000 UTC m=+0.599056267 container cleanup f64ae5e500fe7de359b0f1a4591855660914759ea304b93831780c7aa13dfaab (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:28.875 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 bash[49044]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:28:28.875 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 bash[49070]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:28:29.143 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 podman[49056]: 2024-02-14 17:28:28.874602676 +0000 UTC m=+0.625864162 container remove f64ae5e500fe7de359b0f1a4591855660914759ea304b93831780c7aa13dfaab (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:29.143 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-14T17:28:29.144 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 systemd[1]: Stopped Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:29.144 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Consumed 1.966s CPU time. 2024-02-14T17:28:29.144 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:28 smithi111 systemd[1]: Starting Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:29.144 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 podman[49142]: 2024-02-14 17:28:29.010160094 +0000 UTC m=+0.018465612 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-14T17:28:29.144 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 podman[49142]: 2024-02-14 17:28:29.142375502 +0000 UTC m=+0.150681061 container create 79a5373dc7218b0f31bd1001fdfaaa34812227d76c1301783262b408959bee93 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:29.411 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2583408852' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:28:29.412 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1495433136' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]: dispatch 2024-02-14T17:28:29.412 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]: dispatch 2024-02-14T17:28:29.412 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 podman[49142]: 2024-02-14 17:28:29.319475495 +0000 UTC m=+0.327781012 container init 79a5373dc7218b0f31bd1001fdfaaa34812227d76c1301783262b408959bee93 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:29.412 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 podman[49142]: 2024-02-14 17:28:29.322153162 +0000 UTC m=+0.330458684 container start 79a5373dc7218b0f31bd1001fdfaaa34812227d76c1301783262b408959bee93 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:28:29.412 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.345Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-14T17:28:29.412 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.345Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-14T17:28:29.412 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.345Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-14T17:28:29.412 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.346Z caller=main.go:518 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi111 (none))" 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.346Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.346Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.347Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.347Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.348Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.350Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.350Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.613µs 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.350Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:28:29.413 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 bash[49142]: 79a5373dc7218b0f31bd1001fdfaaa34812227d76c1301783262b408959bee93 2024-02-14T17:28:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:29 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2583408852' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:28:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:29 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1495433136' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]: dispatch 2024-02-14T17:28:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:29 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]: dispatch 2024-02-14T17:28:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:29 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2583408852' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:28:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:29 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1495433136' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]: dispatch 2024-02-14T17:28:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:29 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]: dispatch 2024-02-14T17:28:29.734 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 systemd[1]: Started Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:29.734 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.554Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-02-14T17:28:30.118 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:29 smithi111 systemd[1]: Stopping Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:30.118 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:29 smithi111 bash[49264]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter.b" found: no such container 2024-02-14T17:28:30.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.776Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-02-14T17:28:30.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.776Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-02-14T17:28:30.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.776Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.468µs wal_replay_duration=425.514851ms total_replay_duration=425.560552ms 2024-02-14T17:28:30.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.780Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:28:30.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.780Z caller=main.go:947 level=info msg="TSDB started" 2024-02-14T17:28:30.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.781Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:28:30.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.806Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.370946ms db_storage=983ns remote_storage=1.558µs web_handler=452ns query_engine=930ns scrape=535.596µs scrape_sd=107.155µs notify=20.114µs notify_sd=29.344µs rules=24.339488ms 2024-02-14T17:28:30.119 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:29.806Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-14T17:28:30.409 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 podman[49277]: 2024-02-14 17:28:30.117224558 +0000 UTC m=+0.124468948 container died 8859571f4a9e85cbd8b2b6bb9f9e9d9b32058bbc7639e966b784dc78558be397 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:28:30.409 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:30.409 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]': finished 2024-02-14T17:28:30.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: osdmap e95: 8 total, 8 up, 8 in 2024-02-14T17:28:30.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:30.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:30.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-14T17:28:30.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: Reconfiguring daemon node-exporter.b on smithi111 2024-02-14T17:28:30.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3439380412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]: dispatch 2024-02-14T17:28:30.410 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:30 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]: dispatch 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]': finished 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: osdmap e95: 8 total, 8 up, 8 in 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: Reconfiguring daemon node-exporter.b on smithi111 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3439380412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]: dispatch 2024-02-14T17:28:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]: dispatch 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3178045652"}]': finished 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: osdmap e95: 8 total, 8 up, 8 in 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: Reconfiguring node-exporter.b (dependencies changed)... 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: Reconfiguring daemon node-exporter.b on smithi111 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3439380412' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]: dispatch 2024-02-14T17:28:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:30 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]: dispatch 2024-02-14T17:28:30.732 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 podman[49277]: 2024-02-14 17:28:30.498958351 +0000 UTC m=+0.506202736 container cleanup 8859571f4a9e85cbd8b2b6bb9f9e9d9b32058bbc7639e966b784dc78558be397 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:28:30.732 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 bash[49277]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b 2024-02-14T17:28:30.732 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 bash[49304]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter.b" found: no such container 2024-02-14T17:28:30.984 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 podman[49291]: 2024-02-14 17:28:30.758737265 +0000 UTC m=+0.640969823 container remove 8859571f4a9e85cbd8b2b6bb9f9e9d9b32058bbc7639e966b784dc78558be397 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:28:31.005 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:28:31.005 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-14T17:28:31.005 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 systemd[1]: Stopped Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:31.005 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Consumed 3.877s CPU time. 2024-02-14T17:28:31.005 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 systemd[1]: Starting Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 podman[49383]: 2024-02-14 17:28:30.892304008 +0000 UTC m=+0.016757938 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:30 smithi111 podman[49383]: 2024-02-14 17:28:30.992993024 +0000 UTC m=+0.117446938 container create 5d05f9239e5aff5845bbcb9398dceadd3475555596b4619a47db0a8f6b4d9542 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 podman[49383]: 2024-02-14 17:28:31.212142344 +0000 UTC m=+0.336596263 container init 5d05f9239e5aff5845bbcb9398dceadd3475555596b4619a47db0a8f6b4d9542 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 podman[49383]: 2024-02-14 17:28:31.215505784 +0000 UTC m=+0.339959706 container start 5d05f9239e5aff5845bbcb9398dceadd3475555596b4619a47db0a8f6b4d9542 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.218Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.218Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-02-14T17:28:31.305 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=arp 2024-02-14T17:28:31.306 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=bcache 2024-02-14T17:28:31.306 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=bonding 2024-02-14T17:28:31.306 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=btrfs 2024-02-14T17:28:31.306 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=conntrack 2024-02-14T17:28:31.306 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=cpu 2024-02-14T17:28:31.306 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-02-14T17:28:31.306 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=diskstats 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=dmi 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=edac 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=entropy 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=filefd 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=filesystem 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=hwmon 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=infiniband 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=ipvs 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=loadavg 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=mdadm 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=meminfo 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=netclass 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=netdev 2024-02-14T17:28:31.307 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=netstat 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=nfs 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.219Z caller=node_exporter.go:115 level=info collector=nfsd 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=nvme 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=os 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=pressure 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=rapl 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=schedstat 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=sockstat 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=softnet 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=stat 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=tapestats 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=textfile 2024-02-14T17:28:31.308 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=time 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=uname 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=vmstat 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=xfs 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:115 level=info collector=zfs 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[49400]: ts=2024-02-14T17:28:31.220Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-02-14T17:28:31.309 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 bash[49383]: 5d05f9239e5aff5845bbcb9398dceadd3475555596b4619a47db0a8f6b4d9542 2024-02-14T17:28:31.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:28:31 smithi111 systemd[1]: Started Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]': finished 2024-02-14T17:28:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: osdmap e96: 8 total, 8 up, 8 in 2024-02-14T17:28:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4242561374' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/3428894855"}]: dispatch 2024-02-14T17:28:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:31 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]': finished 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: osdmap e96: 8 total, 8 up, 8 in 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4242561374' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/3428894855"}]: dispatch 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3842588073"}]': finished 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: osdmap e96: 8 total, 8 up, 8 in 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4242561374' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/3428894855"}]: dispatch 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:31.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:31 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:28:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:28:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: Reconfiguring grafana.a (dependencies changed)... 2024-02-14T17:28:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-14T17:28:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:28:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: Reconfiguring daemon grafana.a on smithi111 2024-02-14T17:28:32.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4242561374' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/3428894855"}]': finished 2024-02-14T17:28:32.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: osdmap e97: 8 total, 8 up, 8 in 2024-02-14T17:28:32.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/779328358' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]: dispatch 2024-02-14T17:28:32.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:32 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]: dispatch 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: Reconfiguring grafana.a (dependencies changed)... 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: Reconfiguring daemon grafana.a on smithi111 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4242561374' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/3428894855"}]': finished 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: osdmap e97: 8 total, 8 up, 8 in 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/779328358' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]: dispatch 2024-02-14T17:28:32.842 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]: dispatch 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: Reconfiguring grafana.a (dependencies changed)... 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: Reconfiguring daemon grafana.a on smithi111 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4242561374' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/3428894855"}]': finished 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: osdmap e97: 8 total, 8 up, 8 in 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/779328358' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]: dispatch 2024-02-14T17:28:32.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:32 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]: dispatch 2024-02-14T17:28:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:33 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]': finished 2024-02-14T17:28:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:33 smithi111 ceph-mon[25603]: osdmap e98: 8 total, 8 up, 8 in 2024-02-14T17:28:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:33 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/178061160' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/3428894855"}]: dispatch 2024-02-14T17:28:33.855 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:33 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]': finished 2024-02-14T17:28:33.855 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:33 smithi017 ceph-mon[24769]: osdmap e98: 8 total, 8 up, 8 in 2024-02-14T17:28:33.855 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:33 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/178061160' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/3428894855"}]: dispatch 2024-02-14T17:28:33.856 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:33 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4251098973"}]': finished 2024-02-14T17:28:33.856 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:33 smithi017 ceph-mon[29506]: osdmap e98: 8 total, 8 up, 8 in 2024-02-14T17:28:33.856 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:33 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/178061160' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/3428894855"}]: dispatch 2024-02-14T17:28:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[24769]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/178061160' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/3428894855"}]': finished 2024-02-14T17:28:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[24769]: osdmap e99: 8 total, 8 up, 8 in 2024-02-14T17:28:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/94779526' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]: dispatch 2024-02-14T17:28:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]: dispatch 2024-02-14T17:28:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[29506]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/178061160' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/3428894855"}]': finished 2024-02-14T17:28:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[29506]: osdmap e99: 8 total, 8 up, 8 in 2024-02-14T17:28:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/94779526' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]: dispatch 2024-02-14T17:28:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:34 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]: dispatch 2024-02-14T17:28:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:34 smithi111 ceph-mon[25603]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:34 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/178061160' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/3428894855"}]': finished 2024-02-14T17:28:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:34 smithi111 ceph-mon[25603]: osdmap e99: 8 total, 8 up, 8 in 2024-02-14T17:28:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:34 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/94779526' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]: dispatch 2024-02-14T17:28:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:34 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]: dispatch 2024-02-14T17:28:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:35 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]': finished 2024-02-14T17:28:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:35 smithi111 ceph-mon[25603]: osdmap e100: 8 total, 8 up, 8 in 2024-02-14T17:28:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:35 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2823925979' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1369078824"}]: dispatch 2024-02-14T17:28:35.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:35 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]': finished 2024-02-14T17:28:35.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:35 smithi017 ceph-mon[24769]: osdmap e100: 8 total, 8 up, 8 in 2024-02-14T17:28:35.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:35 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2823925979' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1369078824"}]: dispatch 2024-02-14T17:28:35.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:35 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/689187119"}]': finished 2024-02-14T17:28:35.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:35 smithi017 ceph-mon[29506]: osdmap e100: 8 total, 8 up, 8 in 2024-02-14T17:28:35.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2823925979' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1369078824"}]: dispatch 2024-02-14T17:28:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:36 smithi111 ceph-mon[25603]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:36 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2823925979' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1369078824"}]': finished 2024-02-14T17:28:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:36 smithi111 ceph-mon[25603]: osdmap e101: 8 total, 8 up, 8 in 2024-02-14T17:28:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:36 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[24769]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2823925979' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1369078824"}]': finished 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[24769]: osdmap e101: 8 total, 8 up, 8 in 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[29506]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2823925979' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1369078824"}]': finished 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[29506]: osdmap e101: 8 total, 8 up, 8 in 2024-02-14T17:28:36.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:36 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:37] "GET /metrics HTTP/1.1" 200 37889 "" "Prometheus/2.33.4" 2024-02-14T17:28:38.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:38 smithi017 ceph-mon[24769]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:28:38.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:38 smithi017 ceph-mon[29506]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:28:38.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:38 smithi111 ceph-mon[25603]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:28:38.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:28:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:39 smithi017 ceph-mon[29506]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:39.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:39 smithi017 ceph-mon[24769]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:39.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:39 smithi111 ceph-mon[25603]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:40.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:40 smithi017 ceph-mon[24769]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:40.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:40 smithi017 ceph-mon[29506]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:40.982 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:40 smithi111 ceph-mon[25603]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:42.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:42 smithi017 ceph-mon[24769]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-02-14T17:28:42.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:42 smithi017 ceph-mon[29506]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-02-14T17:28:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:42 smithi111 ceph-mon[25603]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 658 B/s rd, 0 op/s 2024-02-14T17:28:44.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:44 smithi017 ceph-mon[24769]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:28:44.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:44 smithi017 ceph-mon[29506]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:28:44.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:44 smithi111 ceph-mon[25603]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:28:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:46 smithi017 ceph-mon[24769]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:28:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:46 smithi017 ceph-mon[29506]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:28:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:46 smithi111 ceph-mon[25603]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:28:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:47] "GET /metrics HTTP/1.1" 200 37889 "" "Prometheus/2.33.4" 2024-02-14T17:28:48.659 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:48 smithi111 ceph-mon[25603]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-02-14T17:28:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:48 smithi017 ceph-mon[24769]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-02-14T17:28:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:48 smithi017 ceph-mon[29506]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-02-14T17:28:48.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:28:49.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:49 smithi017 ceph-mon[24769]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:49.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:49 smithi017 ceph-mon[29506]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:49 smithi111 ceph-mon[25603]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:28:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:50 smithi017 ceph-mon[24769]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:50 smithi017 ceph-mon[29506]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:50 smithi111 ceph-mon[25603]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:52 smithi017 ceph-mon[24769]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:52 smithi017 ceph-mon[29506]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:52 smithi111 ceph-mon[25603]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:28:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:28:54.350 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:54 smithi111 ceph-mon[25603]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:54 smithi017 ceph-mon[29506]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:54 smithi017 ceph-mon[24769]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:28:55.734 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 systemd[1]: Stopping Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:55.735 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 bash[49609]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana.a" found: no such container 2024-02-14T17:28:55.735 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:28:55+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-14T17:28:55.735 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[41726]: t=2024-02-14T17:28:55+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-02-14T17:28:55.735 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 podman[49620]: 2024-02-14 17:28:55.546774791 +0000 UTC m=+0.123316713 container died 2f503a5ea1563bffbf0a354799d3a509bd661480c0a148139f625e8cef40cf72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.tags=base rhel8, name=ubi8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, description=Ceph Grafana Container, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, version=8.5, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, architecture=x86_64, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=) 2024-02-14T17:28:56.127 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 podman[49620]: 2024-02-14 17:28:55.9010323 +0000 UTC m=+0.477574230 container cleanup 2f503a5ea1563bffbf0a354799d3a509bd661480c0a148139f625e8cef40cf72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.component=ubi8-container, version=8.5, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, release=236.1648460182, io.openshift.tags=base rhel8, name=ubi8, distribution-scope=public, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., architecture=x86_64) 2024-02-14T17:28:56.127 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 bash[49620]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a 2024-02-14T17:28:56.127 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:55 smithi111 bash[49649]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana.a" found: no such container 2024-02-14T17:28:56.399 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-mon[25603]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:28:56.400 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 podman[49635]: 2024-02-14 17:28:56.169173074 +0000 UTC m=+0.624038561 container remove 2f503a5ea1563bffbf0a354799d3a509bd661480c0a148139f625e8cef40cf72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, version=8.5, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, description=Ceph Grafana Container) 2024-02-14T17:28:56.400 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-14T17:28:56.400 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 systemd[1]: Stopped Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:56.400 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service: Consumed 3.407s CPU time. 2024-02-14T17:28:56.400 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 systemd[1]: Starting Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:28:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:56 smithi017 ceph-mon[24769]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:28:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:56 smithi017 ceph-mon[29506]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:28:56.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 podman[49716]: 2024-02-14 17:28:56.301031681 +0000 UTC m=+0.017141432 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-02-14T17:28:56.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 podman[49716]: 2024-02-14 17:28:56.43694902 +0000 UTC m=+0.153058813 container create 3af96da5f3f24efcbd60c650cc0952ade4b3f37f528715a8d07f7db04a46b898 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, name=ubi8, io.buildah.version=1.24.2, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-02-14T17:28:56.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 podman[49716]: 2024-02-14 17:28:56.639302403 +0000 UTC m=+0.355412153 container init 3af96da5f3f24efcbd60c650cc0952ade4b3f37f528715a8d07f7db04a46b898 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, architecture=x86_64, io.openshift.expose-services=, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.buildah.version=1.24.2, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, distribution-scope=public, vcs-type=git) 2024-02-14T17:28:56.670 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 podman[49716]: 2024-02-14 17:28:56.641867247 +0000 UTC m=+0.357976999 container start 3af96da5f3f24efcbd60c650cc0952ade4b3f37f528715a8d07f7db04a46b898 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.openshift.expose-services=, version=8.5, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, name=ubi8, com.redhat.component=ubi8-container) 2024-02-14T17:28:56.940 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Bus STOPPING 2024-02-14T17:28:56.940 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:28:56.940 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Bus STOPPED 2024-02-14T17:28:56.940 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Bus STARTING 2024-02-14T17:28:56.940 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Serving on http://:::9283 2024-02-14T17:28:56.940 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Bus STARTED 2024-02-14T17:28:56.940 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Bus STOPPING 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-02-14T17:28:56.941 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="App mode production" logger=settings 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=506.805µs 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 bash[49716]: 3af96da5f3f24efcbd60c650cc0952ade4b3f37f528715a8d07f7db04a46b898 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 systemd[1]: Started Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:28:56.942 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-02-14T17:28:56.943 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-14T17:28:56.943 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-02-14T17:28:56.943 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-02-14T17:28:56.943 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-02-14T17:28:56.943 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:28:56+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-02-14T17:28:57.215 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:28:57.215 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Bus STOPPED 2024-02-14T17:28:57.215 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:56] ENGINE Bus STARTING 2024-02-14T17:28:57.215 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE Serving on http://:::9283 2024-02-14T17:28:57.216 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE Bus STARTED 2024-02-14T17:28:57.216 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE Bus STOPPING 2024-02-14T17:28:57.216 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:28:57.216 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE Bus STOPPED 2024-02-14T17:28:57.216 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE Bus STARTING 2024-02-14T17:28:57.216 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE Serving on http://:::9283 2024-02-14T17:28:57.216 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:28:57] ENGINE Bus STARTED 2024-02-14T17:28:57.885 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:28:57] "GET /metrics HTTP/1.1" 200 37892 "" "Prometheus/2.33.4" 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-02-14T17:28:57.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:57.887 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:28:57.887 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:28:57.887 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-02-14T17:28:57.887 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-02-14T17:28:57.887 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:57.887 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:57 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:28:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:28:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:28:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-02-14T17:28:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:28:58.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi017.front.sepia.ceph.com:9093"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi111.front.sepia.ceph.com:9095"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:28:58.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-02-14T17:28:58.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi111.front.sepia.ceph.com:3000"}]: dispatch 2024-02-14T17:28:58.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:28:58.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:57 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:28:58.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:58 smithi111 ceph-mon[25603]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:28:58.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:28:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:28:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:28:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:58 smithi017 ceph-mon[24769]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:28:59.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:58 smithi017 ceph-mon[29506]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:29:00.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:28:59 smithi017 ceph-mon[24769]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:00.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:28:59 smithi017 ceph-mon[29506]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:28:59 smithi111 ceph-mon[25603]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:01.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:00 smithi017 ceph-mon[24769]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:01.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:00 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:01.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:00 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:00 smithi017 ceph-mon[29506]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:00 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:00 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:00 smithi111 ceph-mon[25603]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:00 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:00 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:02 smithi017 ceph-mon[24769]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:29:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:02 smithi017 ceph-mon[29506]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:29:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:02 smithi111 ceph-mon[25603]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:29:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:04 smithi017 ceph-mon[24769]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:04 smithi017 ceph-mon[29506]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:04 smithi111 ceph-mon[25603]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:07.222 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:07.223 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:29:07.224 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:29:07.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:07.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:29:07.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:29:07.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:07.962 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:29:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:29:07] "GET /metrics HTTP/1.1" 200 37887 "" "Prometheus/2.33.4" 2024-02-14T17:29:08.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:07 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:29:08.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:07 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:29:08.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:07 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:29:08.243 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:07 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:29:08.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:07 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:29:08.243 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:07 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:29:08.963 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:29:08 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:29:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:29:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:08 smithi017 ceph-mon[24769]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:09.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:08 smithi017 ceph-mon[29506]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:09.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:08 smithi111 ceph-mon[25603]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:10.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:09 smithi017 ceph-mon[24769]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:10.232 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:09 smithi017 ceph-mon[29506]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:10.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:09 smithi111 ceph-mon[25603]: from='client.24851 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:11.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:10 smithi017 ceph-mon[24769]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:11.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:10 smithi017 ceph-mon[29506]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:10 smithi111 ceph-mon[25603]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:12.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:12.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:12.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:29:12.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/464807814' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:29:12.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:11 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/464807814' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:29:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:11 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/464807814' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:29:13.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:12 smithi017 ceph-mon[24769]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:13.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:12 smithi017 ceph-mon[29506]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:12 smithi111 ceph-mon[25603]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:14 smithi111 ceph-mon[25603]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:14 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:14 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:14 smithi017 ceph-mon[24769]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:14 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:14 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:14 smithi017 ceph-mon[29506]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:14 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:14 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[24769]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:29:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[29506]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:16.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:16.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:29:16.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:16 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:16 smithi111 ceph-mon[25603]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:16 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:16 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:16 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:29:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:16 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:29:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:16 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:29:17.896 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:29:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:29:17] "GET /metrics HTTP/1.1" 200 37887 "" "Prometheus/2.33.4" 2024-02-14T17:29:18.659 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:18 smithi111 ceph-mon[25603]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:18 smithi017 ceph-mon[24769]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:18 smithi017 ceph-mon[29506]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:18.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:29:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:29:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:29:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:20 smithi017 ceph-mon[24769]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:20 smithi017 ceph-mon[29506]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:20 smithi111 ceph-mon[25603]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:22 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:22 smithi017 ceph-mon[29506]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:22 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:22 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:22 smithi017 ceph-mon[24769]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:22 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:22 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:22 smithi111 ceph-mon[25603]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:22 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:24 smithi017 ceph-mon[24769]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:24 smithi017 ceph-mon[29506]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:24 smithi111 ceph-mon[25603]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:26 smithi017 ceph-mon[24769]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:26 smithi017 ceph-mon[29506]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:26 smithi111 ceph-mon[25603]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:27.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:29:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:29:27] "GET /metrics HTTP/1.1" 200 37886 "" "Prometheus/2.33.4" 2024-02-14T17:29:28.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:28 smithi111 ceph-mon[25603]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:28 smithi017 ceph-mon[24769]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:28 smithi017 ceph-mon[29506]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:28.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:29:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:29:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:29:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:30 smithi017 ceph-mon[24769]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:30 smithi017 ceph-mon[29506]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:30 smithi111 ceph-mon[25603]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:32.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:32 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:32.597 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:32 smithi017 ceph-mon[24769]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:32.597 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:32 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:32.597 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:32 smithi017 ceph-mon[29506]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:32 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:32 smithi111 ceph-mon[25603]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:34 smithi017 ceph-mon[24769]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:34 smithi017 ceph-mon[29506]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:34 smithi111 ceph-mon[25603]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:36 smithi017 ceph-mon[24769]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:36 smithi017 ceph-mon[29506]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:36 smithi111 ceph-mon[25603]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:37.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:37 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:37.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:37 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:37 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:29:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:29:37] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-02-14T17:29:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:38 smithi017 ceph-mon[24769]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:38 smithi017 ceph-mon[29506]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:38 smithi111 ceph-mon[25603]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:38.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:29:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:29:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:29:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:40 smithi017 ceph-mon[24769]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:40 smithi017 ceph-mon[29506]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:40 smithi111 ceph-mon[25603]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:42 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:42 smithi017 ceph-mon[24769]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:42 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:42 smithi017 ceph-mon[29506]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:42 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:42.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:42 smithi111 ceph-mon[25603]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:44 smithi017 ceph-mon[24769]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:44 smithi017 ceph-mon[29506]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:44 smithi111 ceph-mon[25603]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:46 smithi017 ceph-mon[24769]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:46.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:46 smithi017 ceph-mon[29506]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:46 smithi111 ceph-mon[25603]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:29:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:29:47] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-02-14T17:29:48.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:48 smithi111 ceph-mon[25603]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:48 smithi017 ceph-mon[24769]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:48 smithi017 ceph-mon[29506]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:48.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:29:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:29:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:29:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:50 smithi017 ceph-mon[24769]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:50 smithi017 ceph-mon[29506]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:50 smithi111 ceph-mon[25603]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:52 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:52 smithi017 ceph-mon[24769]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:52 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:52 smithi017 ceph-mon[29506]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:52 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:29:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:52 smithi111 ceph-mon[25603]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:29:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:54 smithi017 ceph-mon[24769]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:54 smithi017 ceph-mon[29506]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:54 smithi111 ceph-mon[25603]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:56 smithi017 ceph-mon[24769]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:56 smithi017 ceph-mon[29506]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:56 smithi111 ceph-mon[25603]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:29:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:29:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:29:57] "GET /metrics HTTP/1.1" 200 37890 "" "Prometheus/2.33.4" 2024-02-14T17:29:58.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:29:58 smithi111 ceph-mon[25603]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:29:58 smithi017 ceph-mon[24769]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:29:58 smithi017 ceph-mon[29506]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:29:58.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:29:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:29:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:30:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:00 smithi017 ceph-mon[24769]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:00 smithi017 ceph-mon[24769]: overall HEALTH_OK 2024-02-14T17:30:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:00 smithi017 ceph-mon[29506]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:00 smithi017 ceph-mon[29506]: overall HEALTH_OK 2024-02-14T17:30:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:00 smithi111 ceph-mon[25603]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:00 smithi111 ceph-mon[25603]: overall HEALTH_OK 2024-02-14T17:30:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:02 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:02 smithi017 ceph-mon[24769]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:02 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:02 smithi017 ceph-mon[29506]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:02 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:02 smithi111 ceph-mon[25603]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:04 smithi017 ceph-mon[24769]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:04 smithi017 ceph-mon[29506]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:04 smithi111 ceph-mon[25603]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:06 smithi017 ceph-mon[24769]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:06 smithi017 ceph-mon[29506]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:06 smithi111 ceph-mon[25603]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:07 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:07.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:07 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:07 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:30:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:30:07] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-14T17:30:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:08 smithi017 ceph-mon[24769]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:08 smithi017 ceph-mon[29506]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:08 smithi111 ceph-mon[25603]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:08.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:30:08 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:30:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:30:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:10 smithi017 ceph-mon[24769]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:10 smithi017 ceph-mon[29506]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:10 smithi111 ceph-mon[25603]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:12 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:12 smithi017 ceph-mon[24769]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:12 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:12 smithi017 ceph-mon[29506]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:12 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:12 smithi111 ceph-mon[25603]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:14 smithi017 ceph-mon[24769]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:14 smithi017 ceph-mon[29506]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:14 smithi111 ceph-mon[25603]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:16 smithi017 ceph-mon[24769]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:16 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:30:16.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:16 smithi017 ceph-mon[29506]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:16.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:16 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:30:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:16 smithi111 ceph-mon[25603]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:16 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:30:17.883 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:30:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:30:17] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-14T17:30:18.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:18 smithi017 ceph-mon[24769]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:18 smithi017 ceph-mon[29506]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:18.659 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:18 smithi111 ceph-mon[25603]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:18.972 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:30:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:30:18.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:30:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:30:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:20 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:20.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:30:20.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:30:20.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:20 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:22 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:22 smithi017 ceph-mon[24769]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:22 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:22 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:22 smithi017 ceph-mon[29506]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:22 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:22 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:22 smithi111 ceph-mon[25603]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:22 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:24 smithi017 ceph-mon[24769]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:24 smithi017 ceph-mon[29506]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:24 smithi111 ceph-mon[25603]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:26 smithi017 ceph-mon[24769]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:26.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:26 smithi017 ceph-mon[29506]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:26 smithi111 ceph-mon[25603]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:30:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:30:27] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-02-14T17:30:28.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:28 smithi111 ceph-mon[25603]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:28 smithi017 ceph-mon[24769]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:28 smithi017 ceph-mon[29506]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:28.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:30:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:30:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:30:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:30 smithi017 ceph-mon[24769]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:30 smithi017 ceph-mon[29506]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:30.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:30 smithi111 ceph-mon[25603]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:32.590 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:32 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:32.590 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:32 smithi017 ceph-mon[29506]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:32.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:32 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:32.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:32 smithi017 ceph-mon[24769]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:32 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:32 smithi111 ceph-mon[25603]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:34 smithi017 ceph-mon[24769]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:34 smithi017 ceph-mon[29506]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:34 smithi111 ceph-mon[25603]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:36 smithi017 ceph-mon[24769]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:36 smithi017 ceph-mon[29506]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:36 smithi111 ceph-mon[25603]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:37.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:37 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:37.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:37 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:37 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:30:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:30:37] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-02-14T17:30:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:38 smithi017 ceph-mon[24769]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:38 smithi017 ceph-mon[29506]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:38 smithi111 ceph-mon[25603]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:38.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:30:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:30:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:30:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:40 smithi017 ceph-mon[24769]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:40 smithi017 ceph-mon[29506]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:40 smithi111 ceph-mon[25603]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:42 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:42 smithi017 ceph-mon[24769]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:42 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:42 smithi017 ceph-mon[29506]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:42 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:42.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:42 smithi111 ceph-mon[25603]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:44 smithi017 ceph-mon[24769]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:44 smithi017 ceph-mon[29506]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:44 smithi111 ceph-mon[25603]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:46 smithi017 ceph-mon[29506]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:46 smithi017 ceph-mon[24769]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:46 smithi111 ceph-mon[25603]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:30:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:30:47] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-02-14T17:30:48.659 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:48 smithi111 ceph-mon[25603]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:48 smithi017 ceph-mon[24769]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:48 smithi017 ceph-mon[29506]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:48.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:30:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:30:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:30:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:50 smithi017 ceph-mon[24769]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:50 smithi017 ceph-mon[29506]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:50 smithi111 ceph-mon[25603]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:52 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:52 smithi017 ceph-mon[24769]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:52 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:52 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:52.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:52 smithi017 ceph-mon[29506]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:52.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:52 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:52 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:30:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:52 smithi111 ceph-mon[25603]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:30:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:52 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:30:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:54 smithi017 ceph-mon[24769]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:54 smithi017 ceph-mon[29506]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:54 smithi111 ceph-mon[25603]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:54.804 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-14T17:30:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:56 smithi017 ceph-mon[24769]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:56 smithi017 ceph-mon[29506]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:56 smithi111 ceph-mon[25603]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:30:57.229 INFO:teuthology.orchestra.run.smithi017.stdout:Scheduled to redeploy mgr.y on host 'smithi017' 2024-02-14T17:30:57.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:57 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2287017880' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:30:57.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:57 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2287017880' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:30:57.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:57 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2287017880' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-14T17:30:57.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps --refresh' 2024-02-14T17:30:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:30:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:30:57] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-02-14T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[24769]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[24769]: Schedule redeploy daemon mgr.y 2024-02-14T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:30:58.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[24769]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[29506]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:30:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[29506]: Schedule redeploy daemon mgr.y 2024-02-14T17:30:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:30:58.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:30:58 smithi017 ceph-mon[29506]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-mon[25603]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:30:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-mon[25603]: Schedule redeploy daemon mgr.y 2024-02-14T17:30:58.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:30:58.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:30:58.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-mon[25603]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:30:58.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:30:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:30:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 41s ago 10m 17.8M - 0.23.0 ba2b418f427c 556f84c8c0ad 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (2m) 41s ago 10m 44.4M - 8.3.5 dad864ee21e9 3af96da5f3f2 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (110s) 41s ago 9m 50.4M - 3.5 e1d6a67b021e 809820ca9afb 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283 running (5m) 41s ago 14m 504M - 19.0.0-1430-g8727ce93 170846cf6432 2ffaf8e8fceb 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:9283 running (16m) 41s ago 16m 411M - 17.2.0 e1d6a67b021e da74f8a06125 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (16m) 41s ago 16m 56.7M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (15m) 41s ago 15m 48.5M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:31:00.897 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (15m) 41s ago 15m 44.4M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (2m) 41s ago 11m 17.3M - 1.3.1 1dbe0e931976 f9e29bdc31de 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (2m) 41s ago 10m 16.6M - 1.3.1 1dbe0e931976 5d05f9239e5a 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (14m) 41s ago 14m 52.4M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (14m) 41s ago 14m 52.0M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (13m) 41s ago 13m 50.0M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (13m) 41s ago 13m 49.6M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (13m) 41s ago 13m 51.2M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (12m) 41s ago 12m 52.4M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (12m) 41s ago 12m 47.6M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (12m) 41s ago 12m 53.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (2m) 41s ago 11m 64.3M - 2.33.4 514e6a882f6e 79a5373dc721 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (9m) 41s ago 9m 73.8M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:31:00.898 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (9m) 41s ago 9m 74.0M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:31:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:31:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:31:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[29506]: Deploying daemon mgr.y on smithi017 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:31:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:01.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:00 smithi017 ceph-mon[24769]: Deploying daemon mgr.y on smithi017 2024-02-14T17:31:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:31:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:31:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:31:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:31:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:00 smithi111 ceph-mon[25603]: Deploying daemon mgr.y on smithi017 2024-02-14T17:31:01.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:01 smithi017 ceph-mon[24769]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:31:01.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:01 smithi017 ceph-mon[29506]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:31:01.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'sleep 180' 2024-02-14T17:31:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:01 smithi111 ceph-mon[25603]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:31:02.931 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:02 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:02.932 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:02 smithi017 ceph-mon[24769]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:02.932 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:02 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:02.932 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:02 smithi017 ceph-mon[29506]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:02.932 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:02 smithi017 systemd[1]: Stopping Ceph mgr.y for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:31:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:02 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:02 smithi111 ceph-mon[25603]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:04.192 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 podman[62791]: 2024-02-14 17:31:04.116112267 +0000 UTC m=+0.417146546 container died da74f8a06125b51f3b7df6cefce247369c7451c88923d6784f47e8db06c63437 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, com.redhat.component=centos-stream-container, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, distribution-scope=public, architecture=x86_64, build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:31:04.834 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 podman[62791]: 2024-02-14 17:31:04.464845884 +0000 UTC m=+0.765880157 container cleanup da74f8a06125b51f3b7df6cefce247369c7451c88923d6784f47e8db06c63437 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, version=8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-02-14T17:31:04.835 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 bash[62791]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y 2024-02-14T17:31:04.835 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 podman[62811]: 2024-02-14 17:31:04.674731526 +0000 UTC m=+0.641779419 container remove da74f8a06125b51f3b7df6cefce247369c7451c88923d6784f47e8db06c63437 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-14T17:31:04.835 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:31:05.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:04 smithi017 ceph-mon[24769]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:05.184 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:04 smithi017 ceph-mon[29506]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:05.184 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-02-14T17:31:05.185 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 systemd[1]: Stopped Ceph mgr.y for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:31:05.185 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:04 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service: Consumed 1min 1.617s CPU time. 2024-02-14T17:31:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:04 smithi111 ceph-mon[25603]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:05.435 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 systemd[1]: Starting Ceph mgr.y for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:31:05.436 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 podman[62943]: 2024-02-14 17:31:05.301451555 +0000 UTC m=+0.018690025 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:31:05.436 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 podman[62943]: 2024-02-14 17:31:05.434737509 +0000 UTC m=+0.151975998 container create bece6d782a35ada8b7e774c6a6ed77384144475e8ed0083c461768073c81a08e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-02-14T17:31:05.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 podman[62943]: 2024-02-14 17:31:05.67932563 +0000 UTC m=+0.396564092 container init bece6d782a35ada8b7e774c6a6ed77384144475e8ed0083c461768073c81a08e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:31:05.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 podman[62943]: 2024-02-14 17:31:05.681957541 +0000 UTC m=+0.399195994 container start bece6d782a35ada8b7e774c6a6ed77384144475e8ed0083c461768073c81a08e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-14T17:31:06.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 ceph-mgr[62961]: -- 172.21.15.17:0/946183162 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e01d9ec1a0 con 0x55e01d9c2800 2024-02-14T17:31:06.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 bash[62943]: bece6d782a35ada8b7e774c6a6ed77384144475e8ed0083c461768073c81a08e 2024-02-14T17:31:06.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:05 smithi017 systemd[1]: Started Ceph mgr.y for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:31:06.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:06.254+0000 7f574d1c7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:31:06.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:06.353+0000 7f574d1c7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:31:06.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:06 smithi111 ceph-mon[25603]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:06.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:06.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:06.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:06.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:06 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:06.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:31:06.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:06 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:07.062 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[24769]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:07.062 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.062 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:07.060+0000 7f574d1c7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[29506]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.063 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.064 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.064 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:07.064 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:31:07.064 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:06 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:07.317 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:07.168+0000 7f574d1c7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:31:07.602 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:07.315+0000 7f574d1c7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:31:07.602 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:07.418+0000 7f574d1c7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:31:07.602 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:07.513+0000 7f574d1c7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:31:07.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:31:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:31:07] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-02-14T17:31:07.918 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:07.786+0000 7f574d1c7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:31:07.918 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:07.916+0000 7f574d1c7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:31:08.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:08 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:08.013+0000 7f574d1c7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:31:08.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:08 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:08.131+0000 7f574d1c7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:31:08.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:08 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:08.310+0000 7f574d1c7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:31:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:08 smithi111 ceph-mon[25603]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:08.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:31:08 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:31:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:31:09.142 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:08 smithi017 ceph-mon[24769]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:09.143 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:08 smithi017 ceph-mon[29506]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:09.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:09 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:09.360+0000 7f574d1c7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:31:09.858 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:09 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:09.504+0000 7f574d1c7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:31:09.858 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:09 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:09.856+0000 7f574d1c7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:31:10.172 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:09 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:09.956+0000 7f574d1c7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:31:10.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:10.170+0000 7f574d1c7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:31:10.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:10.366+0000 7f574d1c7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:31:10.937 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:10.670+0000 7f574d1c7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:10.938 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:10.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:10.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:31:10.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:11.188 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:10.935+0000 7f574d1c7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:31:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:11.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:11.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:31:11.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:31:11.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:10 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:31:11.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:11.185+0000 7f574d1c7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:31:12.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:12.063+0000 7f574d1c7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:31:12.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:12.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-mon[24769]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:12.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:12.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-mon[29506]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:12.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:12.654+0000 7f574d1c7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:31:12.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:12.759+0000 7f574d1c7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:31:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:12 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:13.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:12 smithi111 ceph-mon[25603]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 77 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:13.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:12.990+0000 7f574d1c7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:31:13.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:31:13] ENGINE Bus STARTING 2024-02-14T17:31:13.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: CherryPy Checker: 2024-02-14T17:31:13.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: The Application mounted at '' has an empty config. 2024-02-14T17:31:13.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:31:13.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:31:13] ENGINE Serving on http://:::9283 2024-02-14T17:31:13.441 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:31:13] ENGINE Bus STARTED 2024-02-14T17:31:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[29506]: Standby manager daemon y restarted 2024-02-14T17:31:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[29506]: Standby manager daemon y started 2024-02-14T17:31:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:31:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:31:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:31:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:31:14.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[24769]: Standby manager daemon y restarted 2024-02-14T17:31:14.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[24769]: Standby manager daemon y started 2024-02-14T17:31:14.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:31:14.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:31:14.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:31:14.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:13 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:31:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:13 smithi111 ceph-mon[25603]: Standby manager daemon y restarted 2024-02-14T17:31:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:13 smithi111 ceph-mon[25603]: Standby manager daemon y started 2024-02-14T17:31:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:13 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:31:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:13 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:31:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:13 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:31:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:13 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:31:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:14 smithi017 ceph-mon[24769]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-14T17:31:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:14 smithi017 ceph-mon[24769]: mgrmap e27: x(active, since 3m), standbys: y 2024-02-14T17:31:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:14 smithi017 ceph-mon[29506]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-14T17:31:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:14 smithi017 ceph-mon[29506]: mgrmap e27: x(active, since 3m), standbys: y 2024-02-14T17:31:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:14 smithi111 ceph-mon[25603]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-14T17:31:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:14 smithi111 ceph-mon[25603]: mgrmap e27: x(active, since 3m), standbys: y 2024-02-14T17:31:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:16 smithi017 ceph-mon[24769]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-14T17:31:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:16 smithi017 ceph-mon[29506]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-14T17:31:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:16 smithi111 ceph-mon[25603]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-02-14T17:31:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:31:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:31:17] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-02-14T17:31:18.659 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:18 smithi111 ceph-mon[25603]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-14T17:31:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:18 smithi017 ceph-mon[24769]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-14T17:31:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:18 smithi017 ceph-mon[29506]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-02-14T17:31:18.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:31:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:31:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:31:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:20 smithi017 ceph-mon[24769]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:20 smithi017 ceph-mon[29506]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:20 smithi111 ceph-mon[25603]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:22 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:22 smithi017 ceph-mon[29506]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:22 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:22 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:22 smithi017 ceph-mon[24769]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:22 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:22 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:22 smithi111 ceph-mon[25603]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:22 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:24 smithi017 ceph-mon[24769]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:24 smithi017 ceph-mon[29506]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:24 smithi111 ceph-mon[25603]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:31:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:26 smithi017 ceph-mon[24769]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:26 smithi017 ceph-mon[29506]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:26 smithi111 ceph-mon[25603]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:31:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:31:27] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-02-14T17:31:28.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:28 smithi111 ceph-mon[25603]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:28 smithi017 ceph-mon[24769]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:28 smithi017 ceph-mon[29506]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:28.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:31:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:31:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:31:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:30 smithi017 ceph-mon[29506]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:30 smithi017 ceph-mon[24769]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:30 smithi111 ceph-mon[25603]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:32.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:32 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:32.610 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:32 smithi017 ceph-mon[24769]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:32.610 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:32 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:32.610 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:32 smithi017 ceph-mon[29506]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:32 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:32 smithi111 ceph-mon[25603]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:34 smithi017 ceph-mon[24769]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:34 smithi017 ceph-mon[29506]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:34 smithi111 ceph-mon[25603]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:36 smithi017 ceph-mon[24769]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:36.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:36 smithi017 ceph-mon[29506]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:36 smithi111 ceph-mon[25603]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:37.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:37 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:37.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:37 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:37 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:31:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:31:37] "GET /metrics HTTP/1.1" 200 37892 "" "Prometheus/2.33.4" 2024-02-14T17:31:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:38 smithi017 ceph-mon[24769]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:38 smithi017 ceph-mon[29506]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:38 smithi111 ceph-mon[25603]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:38.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:31:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:31:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:31:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:40 smithi017 ceph-mon[24769]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:40 smithi017 ceph-mon[29506]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:40 smithi111 ceph-mon[25603]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:42 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:42 smithi017 ceph-mon[24769]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:42 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:42 smithi017 ceph-mon[29506]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:42 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:42 smithi111 ceph-mon[25603]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:44 smithi017 ceph-mon[29506]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:44 smithi017 ceph-mon[24769]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:44 smithi111 ceph-mon[25603]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:46 smithi017 ceph-mon[24769]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:46.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:46 smithi017 ceph-mon[29506]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:46 smithi111 ceph-mon[25603]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:31:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:31:47] "GET /metrics HTTP/1.1" 200 37892 "" "Prometheus/2.33.4" 2024-02-14T17:31:48.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:48 smithi111 ceph-mon[25603]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:48 smithi017 ceph-mon[24769]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:48 smithi017 ceph-mon[29506]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:48.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:31:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:31:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:31:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:50 smithi017 ceph-mon[24769]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:50 smithi017 ceph-mon[29506]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:50 smithi111 ceph-mon[25603]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:51 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:51.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:51 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:51 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:31:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:52 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:52 smithi017 ceph-mon[29506]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:52 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:52 smithi017 ceph-mon[24769]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:52 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:31:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:52 smithi111 ceph-mon[25603]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:54 smithi017 ceph-mon[24769]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:54 smithi017 ceph-mon[29506]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:54 smithi111 ceph-mon[25603]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:56 smithi017 ceph-mon[24769]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:56 smithi017 ceph-mon[29506]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:56 smithi111 ceph-mon[25603]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:31:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:31:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:31:57] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-02-14T17:31:58.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:31:58 smithi111 ceph-mon[25603]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:31:58 smithi017 ceph-mon[24769]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:31:58 smithi017 ceph-mon[29506]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:31:58.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:31:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:31:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:32:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:00 smithi017 ceph-mon[24769]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:00 smithi017 ceph-mon[29506]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:00.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:00 smithi111 ceph-mon[25603]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:02 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:02 smithi017 ceph-mon[29506]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:02 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:02 smithi017 ceph-mon[24769]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:02 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:02 smithi111 ceph-mon[25603]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:04 smithi017 ceph-mon[29506]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:04 smithi017 ceph-mon[24769]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:04 smithi111 ceph-mon[25603]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:06 smithi017 ceph-mon[29506]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:06 smithi017 ceph-mon[24769]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:06 smithi111 ceph-mon[25603]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:07 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:07.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:07 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:07 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:32:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:32:07] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-02-14T17:32:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:08 smithi017 ceph-mon[24769]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:08 smithi017 ceph-mon[29506]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:08 smithi111 ceph-mon[25603]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:08.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:32:08 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:32:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:32:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:10 smithi017 ceph-mon[24769]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:10 smithi017 ceph-mon[29506]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:10 smithi111 ceph-mon[25603]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:11.659 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:11 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:32:11.660 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:11 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:32:11.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:11 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:32:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:12 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:12 smithi017 ceph-mon[29506]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:12 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:12 smithi017 ceph-mon[24769]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:12 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:12 smithi111 ceph-mon[25603]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:14.616 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:14 smithi017 ceph-mon[24769]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:14.617 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:14 smithi017 ceph-mon[29506]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:14 smithi111 ceph-mon[25603]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[29506]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:32:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:32:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[24769]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:32:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:32:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:16 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:16 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:16 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:16 smithi111 ceph-mon[25603]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:16 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:32:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:16 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:32:16.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:16 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:32:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:32:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:32:17] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-02-14T17:32:18.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:18 smithi111 ceph-mon[25603]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:18 smithi017 ceph-mon[24769]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:18 smithi017 ceph-mon[29506]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:18.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:32:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:32:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:32:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:20 smithi017 ceph-mon[24769]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:20 smithi017 ceph-mon[29506]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:20 smithi111 ceph-mon[25603]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:21 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:21 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:21.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:21 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:22 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:22 smithi017 ceph-mon[24769]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:22 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:22 smithi017 ceph-mon[29506]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:22 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:22 smithi111 ceph-mon[25603]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:24 smithi017 ceph-mon[24769]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:24 smithi017 ceph-mon[29506]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:24 smithi111 ceph-mon[25603]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:26 smithi017 ceph-mon[24769]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:26 smithi017 ceph-mon[29506]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:26 smithi111 ceph-mon[25603]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:32:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:32:27] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-02-14T17:32:28.666 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:28 smithi111 ceph-mon[25603]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:28 smithi017 ceph-mon[24769]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:28.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:28 smithi017 ceph-mon[29506]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:28.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:32:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:32:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:32:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:30 smithi017 ceph-mon[24769]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:30 smithi017 ceph-mon[29506]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:30 smithi111 ceph-mon[25603]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:32 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:32 smithi017 ceph-mon[29506]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:32 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:32 smithi017 ceph-mon[24769]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:32 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:32 smithi111 ceph-mon[25603]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:34 smithi017 ceph-mon[24769]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:34 smithi017 ceph-mon[29506]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:34 smithi111 ceph-mon[25603]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:36 smithi017 ceph-mon[24769]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:36 smithi017 ceph-mon[29506]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:36.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:36 smithi111 ceph-mon[25603]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:37.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:37 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:37.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:37 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:37 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:32:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:32:37] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-02-14T17:32:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:38 smithi017 ceph-mon[24769]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:38 smithi017 ceph-mon[29506]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:38.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:38 smithi111 ceph-mon[25603]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:38.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:32:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:32:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:32:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:40 smithi017 ceph-mon[24769]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:40 smithi017 ceph-mon[29506]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:40 smithi111 ceph-mon[25603]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:42 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:42 smithi017 ceph-mon[24769]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:42 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:42 smithi017 ceph-mon[29506]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:42 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:42.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:42 smithi111 ceph-mon[25603]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:44 smithi017 ceph-mon[24769]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:44 smithi017 ceph-mon[29506]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:44 smithi111 ceph-mon[25603]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:46.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:46 smithi017 ceph-mon[24769]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:46.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:46 smithi017 ceph-mon[29506]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:46 smithi111 ceph-mon[25603]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:32:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:32:47] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-02-14T17:32:48.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:48 smithi111 ceph-mon[25603]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:48 smithi017 ceph-mon[24769]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:48 smithi017 ceph-mon[29506]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:48.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:32:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:32:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:32:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:50 smithi017 ceph-mon[24769]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:50 smithi017 ceph-mon[29506]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:50 smithi111 ceph-mon[25603]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:51 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:51.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:51 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:51 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:32:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:52 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:52 smithi017 ceph-mon[24769]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:52 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:52 smithi017 ceph-mon[29506]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:52 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:32:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:52 smithi111 ceph-mon[25603]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:54.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:54 smithi017 ceph-mon[24769]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:54.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:54 smithi017 ceph-mon[29506]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:54.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:54 smithi111 ceph-mon[25603]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:56.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:56 smithi017 ceph-mon[24769]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:56.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:56 smithi017 ceph-mon[29506]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:56 smithi111 ceph-mon[25603]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:32:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:32:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:32:57] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-02-14T17:32:58.660 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:32:58 smithi111 ceph-mon[25603]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:32:58 smithi017 ceph-mon[24769]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:32:58 smithi017 ceph-mon[29506]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:32:58.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:32:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:32:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:33:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:00 smithi017 ceph-mon[24769]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:00 smithi017 ceph-mon[29506]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:00 smithi111 ceph-mon[25603]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:02 smithi017 ceph-mon[24769]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:02 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:02 smithi017 ceph-mon[29506]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:02 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:02 smithi111 ceph-mon[25603]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:02 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:04 smithi017 ceph-mon[24769]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:04 smithi017 ceph-mon[29506]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:04 smithi111 ceph-mon[25603]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:06 smithi017 ceph-mon[24769]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:06 smithi017 ceph-mon[29506]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:06 smithi111 ceph-mon[25603]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:07 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:07.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:07 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:07 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:33:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:33:07] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-02-14T17:33:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:08 smithi017 ceph-mon[24769]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:08 smithi017 ceph-mon[29506]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:08 smithi111 ceph-mon[25603]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:08.985 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:33:08 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:33:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:33:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:10 smithi017 ceph-mon[24769]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:10 smithi017 ceph-mon[29506]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:10 smithi111 ceph-mon[25603]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:12 smithi017 ceph-mon[24769]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:12 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:12 smithi017 ceph-mon[29506]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:12 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:12 smithi111 ceph-mon[25603]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:12 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:14 smithi017 ceph-mon[24769]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:14 smithi017 ceph-mon[29506]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:14 smithi111 ceph-mon[25603]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:16 smithi017 ceph-mon[24769]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:16 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:33:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:16 smithi017 ceph-mon[29506]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:16 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:33:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:16 smithi111 ceph-mon[25603]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:16 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:33:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:17 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:33:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:17 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:33:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:17 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:33:17.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:17 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:33:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:17 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:33:17.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:17 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:33:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:17 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:33:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:17 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:33:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:17 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:33:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:33:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:33:17] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-02-14T17:33:18.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:18 smithi017 ceph-mon[24769]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:18 smithi017 ceph-mon[29506]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:18.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:18 smithi111 ceph-mon[25603]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:18.984 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:33:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:33:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"smithi111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.17\", device_class=\"ssd\", hostname=\"smithi017\", instance=\"172.21.15.111:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.17\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-02-14T17:33:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:20 smithi017 ceph-mon[24769]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:20 smithi017 ceph-mon[29506]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:20 smithi111 ceph-mon[25603]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:21 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:21 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:21.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:21 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:22 smithi017 ceph-mon[24769]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:22 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:22 smithi017 ceph-mon[29506]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:22 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:22 smithi111 ceph-mon[25603]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:22 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:24 smithi017 ceph-mon[24769]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:24 smithi017 ceph-mon[29506]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:24 smithi111 ceph-mon[25603]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:26 smithi017 ceph-mon[29506]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:26 smithi017 ceph-mon[24769]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:26 smithi111 ceph-mon[25603]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:33:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:33:27] "GET /metrics HTTP/1.1" 200 37892 "" "Prometheus/2.33.4" 2024-02-14T17:33:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:28 smithi017 ceph-mon[24769]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:28 smithi017 ceph-mon[29506]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:28 smithi111 ceph-mon[25603]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:30 smithi017 ceph-mon[29506]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:30 smithi017 ceph-mon[24769]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:30 smithi111 ceph-mon[25603]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:32 smithi017 ceph-mon[24769]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:32 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:32.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:32 smithi017 ceph-mon[29506]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:32.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:32 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:32 smithi111 ceph-mon[25603]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:32 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:34 smithi017 ceph-mon[29506]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:34 smithi017 ceph-mon[24769]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:34 smithi111 ceph-mon[25603]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:36 smithi111 ceph-mon[25603]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:36 smithi017 ceph-mon[24769]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:36 smithi017 ceph-mon[29506]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:37.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:37 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:37 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:37.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:37 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:33:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:33:37] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-02-14T17:33:38.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:38 smithi111 ceph-mon[25603]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:38 smithi017 ceph-mon[24769]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:38.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:38 smithi017 ceph-mon[29506]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:40 smithi017 ceph-mon[24769]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:40 smithi017 ceph-mon[29506]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:40 smithi111 ceph-mon[25603]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:42 smithi017 ceph-mon[24769]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:42 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:42 smithi017 ceph-mon[29506]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:42 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:42 smithi111 ceph-mon[25603]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:42 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:44 smithi017 ceph-mon[24769]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:44 smithi017 ceph-mon[29506]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:44 smithi111 ceph-mon[25603]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:46 smithi111 ceph-mon[25603]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:46 smithi017 ceph-mon[24769]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:46.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:46 smithi017 ceph-mon[29506]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:33:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:33:47] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-02-14T17:33:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:48 smithi017 ceph-mon[24769]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:48 smithi017 ceph-mon[29506]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:48 smithi111 ceph-mon[25603]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:50 smithi017 ceph-mon[24769]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:50 smithi017 ceph-mon[29506]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:50 smithi111 ceph-mon[25603]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:51 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:51.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:51 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:51 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:33:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:52 smithi017 ceph-mon[29506]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:52 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:52 smithi017 ceph-mon[24769]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:52 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:52 smithi111 ceph-mon[25603]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:52 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:33:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:54 smithi017 ceph-mon[24769]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:54 smithi017 ceph-mon[29506]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:54.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:54 smithi111 ceph-mon[25603]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:56 smithi111 ceph-mon[25603]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:56 smithi017 ceph-mon[24769]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:56 smithi017 ceph-mon[29506]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:33:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:33:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:33:57] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-02-14T17:33:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:33:58 smithi017 ceph-mon[24769]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:33:58 smithi017 ceph-mon[29506]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:33:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:33:58 smithi111 ceph-mon[25603]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:00 smithi017 ceph-mon[29506]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:00 smithi017 ceph-mon[24769]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:00 smithi111 ceph-mon[25603]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:02 smithi017 ceph-mon[29506]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:02.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:02 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:02.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:02 smithi017 ceph-mon[24769]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:02.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:02 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:02 smithi111 ceph-mon[25603]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:02 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:04 smithi017 ceph-mon[24769]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:04 smithi017 ceph-mon[29506]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:04 smithi111 ceph-mon[25603]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:04.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:34:06.428 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:06 smithi017 ceph-mon[24769]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:06.429 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:06 smithi017 ceph-mon[29506]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:06 smithi111 ceph-mon[25603]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:07.208 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (5m) 112s ago 13m 25.4M - 0.23.0 ba2b418f427c 556f84c8c0ad 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (5m) 2m ago 13m 53.3M - 8.3.5 dad864ee21e9 3af96da5f3f2 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (4m) 112s ago 12m 52.9M - 3.5 e1d6a67b021e 809820ca9afb 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283 running (8m) 2m ago 18m 506M - 19.0.0-1430-g8727ce93 170846cf6432 2ffaf8e8fceb 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (3m) 112s ago 19m 450M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (19m) 112s ago 19m 54.9M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (18m) 2m ago 18m 49.9M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (18m) 112s ago 18m 39.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (6m) 112s ago 14m 17.2M - 1.3.1 1dbe0e931976 f9e29bdc31de 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (5m) 2m ago 14m 17.5M - 1.3.1 1dbe0e931976 5d05f9239e5a 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (17m) 112s ago 17m 53.3M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (17m) 112s ago 17m 53.1M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:34:07.209 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (17m) 112s ago 17m 51.3M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (16m) 112s ago 16m 50.0M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (16m) 2m ago 16m 51.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (15m) 2m ago 15m 52.3M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (15m) 2m ago 15m 47.8M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (15m) 2m ago 15m 53.1M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (5m) 2m ago 14m 64.3M - 2.33.4 514e6a882f6e 79a5373dc721 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (13m) 112s ago 13m 74.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:34:07.210 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (12m) 2m ago 12m 74.2M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:34:07.415 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:07 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:07.415 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:07 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:07 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:07.879 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions' 2024-02-14T17:34:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:34:07] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-02-14T17:34:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:08 smithi111 ceph-mon[25603]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:34:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:08 smithi111 ceph-mon[25603]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:08.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:08 smithi017 ceph-mon[24769]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:34:08.526 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:08 smithi017 ceph-mon[24769]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:08.526 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:08 smithi017 ceph-mon[29506]: from='client.15180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:34:08.526 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:08 smithi017 ceph-mon[29506]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:10.006 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:34:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:34:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:34:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:34:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:34:10.006 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:34:10.007 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:34:10.661 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:10 smithi017 ceph-mon[24769]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:10.662 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:10 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1749543653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:34:10.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:10 smithi017 ceph-mon[29506]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:10.662 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:10 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1749543653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:34:10.684 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:34:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:10 smithi111 ceph-mon[25603]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:10 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1749543653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:34:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:12 smithi017 ceph-mon[24769]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:12 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:12.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:12 smithi017 ceph-mon[29506]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:12.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:12 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:12 smithi111 ceph-mon[25603]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:12 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:12.886 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-02-14T17:34:13.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:13 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3667273510' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:34:13.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:13 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3667273510' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:34:13.511 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph -s' 2024-02-14T17:34:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:13 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3667273510' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:34:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:14 smithi017 ceph-mon[24769]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:14 smithi017 ceph-mon[29506]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:14 smithi111 ceph-mon[25603]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:15.530 INFO:teuthology.orchestra.run.smithi017.stdout: cluster: 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: id: 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: health: HEALTH_OK 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: services: 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: mgr: x(active, since 6m), standbys: y 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: data: 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: pools: 6 pools, 161 pgs 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: objects: 209 objects, 585 KiB 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: usage: 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:15.531 INFO:teuthology.orchestra.run.smithi017.stdout: pgs: 161 active+clean 2024-02-14T17:34:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:34:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: io: 2024-02-14T17:34:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-02-14T17:34:15.532 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:34:16.186 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph mgr fail' 2024-02-14T17:34:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:16 smithi017 ceph-mon[24769]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:16 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3964833615' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:34:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:16 smithi017 ceph-mon[29506]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:16.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:16 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3964833615' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:34:16.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:16 smithi111 ceph-mon[25603]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:34:16.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:16 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3964833615' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:34:17.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:17 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:17.485 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:17 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:17.485 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:17 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:34:17] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-02-14T17:34:18.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:18 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:18.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:18 smithi111 ceph-mon[25603]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:18.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:18 smithi111 ceph-mon[25603]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:34:18.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:18 smithi111 ceph-mon[25603]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[24769]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[24769]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:34:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[24769]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:18.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:18.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[29506]: from='mgr.24785 172.21.15.111:0/3165856438' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:18.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[29506]: from='mgr.24785 ' entity='mgr.x' 2024-02-14T17:34:18.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:18 smithi017 ceph-mon[29506]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:34:19.490 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/967499692' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:34:19.490 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-mon[25603]: osdmap e102: 8 total, 8 up, 8 in 2024-02-14T17:34:19.490 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:19.242+0000 7fa5c267d700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-14T17:34:19.491 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ignoring --setuser ceph since I am not root 2024-02-14T17:34:19.491 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ignoring --setgroup ceph since I am not root 2024-02-14T17:34:19.491 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-mgr[45832]: -- 172.21.15.111:0/443612318 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56168b9961a0 con 0x56168b96cc00 2024-02-14T17:34:19.529 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/967499692' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:34:19.529 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-mon[24769]: osdmap e102: 8 total, 8 up, 8 in 2024-02-14T17:34:19.529 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/967499692' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:34:19.530 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-mon[29506]: osdmap e102: 8 total, 8 up, 8 in 2024-02-14T17:34:19.530 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:34:19] ENGINE Bus STOPPING 2024-02-14T17:34:19.530 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:34:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:34:19.530 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:34:19] ENGINE Bus STOPPED 2024-02-14T17:34:19.530 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:34:19] ENGINE Bus STARTING 2024-02-14T17:34:19.531 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:34:19] ENGINE Serving on http://:::9283 2024-02-14T17:34:19.531 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:19 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:34:19] ENGINE Bus STARTED 2024-02-14T17:34:19.917 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'sleep 180' 2024-02-14T17:34:20.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:19.834+0000 7fe34f6ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:34:20.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:19.931+0000 7fe34f6ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:34:20.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/967499692' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: mgrmap e28: y(active, starting, since 0.95819s) 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:34:20.494 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: Manager daemon y is now available 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[24769]: [14/Feb/2024:17:34:19] ENGINE Bus STARTING 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/967499692' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: mgrmap e28: y(active, starting, since 0.95819s) 2024-02-14T17:34:20.495 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:34:20.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: Manager daemon y is now available 2024-02-14T17:34:20.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:20.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:20.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:20.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:34:20.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:34:20.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:20 smithi017 ceph-mon[29506]: [14/Feb/2024:17:34:19] ENGINE Bus STARTING 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/967499692' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: mgrmap e28: y(active, starting, since 0.95819s) 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:34:20.603 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: Manager daemon y is now available 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:34:20.604 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:34:20.605 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-mon[25603]: [14/Feb/2024:17:34:19] ENGINE Bus STARTING 2024-02-14T17:34:20.856 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:20.616+0000 7fe34f6ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:34:20.856 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:20.709+0000 7fe34f6ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:34:20.856 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:20.854+0000 7fe34f6ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:34:21.199 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:20.956+0000 7fe34f6ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:34:21.199 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:21.046+0000 7fe34f6ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:34:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-mon[25603]: [14/Feb/2024:17:34:19] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:34:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-mon[25603]: [14/Feb/2024:17:34:19] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:34:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-mon[25603]: [14/Feb/2024:17:34:19] ENGINE Bus STARTED 2024-02-14T17:34:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-mon[25603]: mgrmap e29: y(active, since 1.96317s) 2024-02-14T17:34:21.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-mon[25603]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:21.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:21.319+0000 7fe34f6ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:34:21.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:21.434+0000 7fe34f6ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:34:21.501 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[24769]: [14/Feb/2024:17:34:19] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:34:21.501 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[24769]: [14/Feb/2024:17:34:19] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:34:21.501 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[24769]: [14/Feb/2024:17:34:19] ENGINE Bus STARTED 2024-02-14T17:34:21.501 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[24769]: mgrmap e29: y(active, since 1.96317s) 2024-02-14T17:34:21.502 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[24769]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:21.502 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[29506]: [14/Feb/2024:17:34:19] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:34:21.502 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[29506]: [14/Feb/2024:17:34:19] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:34:21.502 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[29506]: [14/Feb/2024:17:34:19] ENGINE Bus STARTED 2024-02-14T17:34:21.502 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[29506]: mgrmap e29: y(active, since 1.96317s) 2024-02-14T17:34:21.502 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:21 smithi017 ceph-mon[29506]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:21.825 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:21.523+0000 7fe34f6ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:34:21.825 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:21.620+0000 7fe34f6ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:34:21.825 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:21.823+0000 7fe34f6ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:34:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:22 smithi111 ceph-mon[25603]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:22 smithi111 ceph-mon[25603]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:22.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:22 smithi017 ceph-mon[24769]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:22.584 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:22 smithi017 ceph-mon[24769]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:22.584 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:22 smithi017 ceph-mon[29506]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:22.584 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:22 smithi017 ceph-mon[29506]: from='client.24908 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:23.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:22.828+0000 7fe34f6ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:34:23.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:22.959+0000 7fe34f6ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:34:23.596 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:23 smithi017 ceph-mon[24769]: mgrmap e30: y(active, since 3s) 2024-02-14T17:34:23.596 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:23 smithi017 ceph-mon[29506]: mgrmap e30: y(active, since 3s) 2024-02-14T17:34:23.617 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:23 smithi111 ceph-mon[25603]: mgrmap e30: y(active, since 3s) 2024-02-14T17:34:23.618 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:23.312+0000 7fe34f6ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:34:23.618 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:23.408+0000 7fe34f6ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:34:23.618 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:23.616+0000 7fe34f6ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:34:23.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:23.808+0000 7fe34f6ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:34:24.382 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:24.109+0000 7fe34f6ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:34:24.383 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:24 smithi111 ceph-mon[25603]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:24.383 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:24 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:24.383 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:24 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:24 smithi017 ceph-mon[24769]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:24 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:24 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:24.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:24 smithi017 ceph-mon[29506]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:24.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:24 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:24.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:24 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:24.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:24.381+0000 7fe34f6ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:34:24.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:24.629+0000 7fe34f6ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:34:25.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:25.472+0000 7fe34f6ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:34:26.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-mon[25603]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:26.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.366 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.366 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:26.045+0000 7fe34f6ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:34:26.366 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:26.144+0000 7fe34f6ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:34:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[29506]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[24769]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:26.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:26 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:26.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:26.363+0000 7fe34f6ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:34:26.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:34:26] ENGINE Bus STARTING 2024-02-14T17:34:26.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CherryPy Checker: 2024-02-14T17:34:26.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: The Application mounted at '' has an empty config. 2024-02-14T17:34:26.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:34:26.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:34:26] ENGINE Serving on http://:::9283 2024-02-14T17:34:26.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:34:26] ENGINE Bus STARTED 2024-02-14T17:34:27.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: Standby manager daemon x started 2024-02-14T17:34:27.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:34:27.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:34:27.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:34:27.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:34:27.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:34:27.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:34:27.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:34:27.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:27.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:27.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:27.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:27.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:34:27.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:34:27.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: Standby manager daemon x started 2024-02-14T17:34:27.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:34:27.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:34:27.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:34:27.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:34:27.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:27.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:27.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:27.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:27.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:34:27.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:27 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:34:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:34:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:34:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:34:28.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:34:28.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:34:28.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:34:28.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:34:28.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:34:28.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:34:28.352 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: mgrmap e31: y(active, since 8s), standbys: x 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:34:28.353 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: mgrmap e31: y(active, since 8s), standbys: x 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:34:28.354 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:34:28.355 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:28.355 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:28 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:34:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: mgrmap e31: y(active, since 8s), standbys: x 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:28.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:34:28.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:34:28.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:28.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:28 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:34:30.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:29.786Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-02-14T17:34:30.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:29.786Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-02-14T17:34:30.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:29.787Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-02-14T17:34:30.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:29.787Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.111:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.111:8765: connect: connection refused" 2024-02-14T17:34:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-mon[24769]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:34:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-mon[29506]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:34:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:30 smithi111 ceph-mon[25603]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:34:30.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:34:30.561+0000 7f56d2ebb700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:30.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Creating ceph-iscsi config... 2024-02-14T17:34:30.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:30.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:30.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:30.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:30.945 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Traceback (most recent call last): 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: return _run_code(code, main_globals, None, 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: exec(code, run_globals) 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:30.946 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Traceback (most recent call last): 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: return future.result(timeout) 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: return self.__get_result() 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: raise self._exception 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: config_blobs=daemon_spec.final_config, 2024-02-14T17:34:30.947 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Creating ceph-iscsi config... 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: Traceback (most recent call last): 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: return _run_code(code, main_globals, None, 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: exec(code, run_globals) 2024-02-14T17:34:30.948 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:30.949 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:30.949 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:30.950 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:30.950 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:30.950 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:30.950 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:30.951 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:30.951 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:31 smithi017 ceph-mon[24769]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:34:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-02-14T17:34:31.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:31.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:31.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:31 smithi017 ceph-mon[29506]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:31.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:34:31.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:31.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:31 smithi017 ceph-mon[24769]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:31 smithi017 ceph-mon[29506]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:31 smithi111 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:31.700 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:34:31.701 INFO:journalctl@ceph.mon.b.smithi111.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: return future.result(timeout) 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.__get_result() 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: raise self._exception 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:34:31.702 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:34:31.703 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:34:31.704 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:34:31.704 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:34:31.704 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:34:31.704 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:34:31.704 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:31 smithi111 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:34:32.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:32 smithi111 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:34:32.579 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:32 smithi111 ceph-mon[25603]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:34:32.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:32 smithi017 ceph-mon[24769]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:34:32.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:32 smithi017 ceph-mon[24769]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:34:32.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:32 smithi017 ceph-mon[29506]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:34:32.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:32 smithi017 ceph-mon[29506]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:34:34.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 systemd[1]: Stopping Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:34:34.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 bash[54652]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:34:34.585 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-mon[25603]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:34.585 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.274Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:34:34.585 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.274Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:34:34.585 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[49157]: ts=2024-02-14T17:34:34.275Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-14T17:34:34.586 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 podman[54665]: 2024-02-14 17:34:34.28499629 +0000 UTC m=+0.152247239 container died 79a5373dc7218b0f31bd1001fdfaaa34812227d76c1301783262b408959bee93 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:34:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:34 smithi017 ceph-mon[24769]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:34 smithi017 ceph-mon[29506]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:34.909 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 podman[54665]: 2024-02-14 17:34:34.690703179 +0000 UTC m=+0.557954104 container cleanup 79a5373dc7218b0f31bd1001fdfaaa34812227d76c1301783262b408959bee93 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:34:34.910 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 bash[54665]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:34:34.910 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 bash[54694]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:34:35.185 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 podman[54679]: 2024-02-14 17:34:34.908543743 +0000 UTC m=+0.624766118 container remove 79a5373dc7218b0f31bd1001fdfaaa34812227d76c1301783262b408959bee93 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:34:35.185 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-14T17:34:35.185 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 systemd[1]: Stopped Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:34:35.185 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Consumed 1.879s CPU time. 2024-02-14T17:34:35.185 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:34 smithi111 systemd[1]: Starting Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:34:35.185 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 podman[54759]: 2024-02-14 17:34:35.046486999 +0000 UTC m=+0.019170668 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-14T17:34:35.186 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 podman[54759]: 2024-02-14 17:34:35.184573557 +0000 UTC m=+0.157257229 container create 877352eda179a0572ce1e1e990a2338630776d0554437a00663d4ff195b14e8b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:34:35.441 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:35 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:35 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 podman[54759]: 2024-02-14 17:34:35.436922862 +0000 UTC m=+0.409606535 container init 877352eda179a0572ce1e1e990a2338630776d0554437a00663d4ff195b14e8b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 podman[54759]: 2024-02-14 17:34:35.439555386 +0000 UTC m=+0.412239056 container start 877352eda179a0572ce1e1e990a2338630776d0554437a00663d4ff195b14e8b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.462Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.462Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.462Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.462Z caller=main.go:518 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi111 (none))" 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.462Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.462Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.464Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:34:35.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.464Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-14T17:34:35.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.465Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-14T17:34:35.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.467Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:34:35.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.467Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.693µs 2024-02-14T17:34:35.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.467Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:34:35.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 bash[54759]: 877352eda179a0572ce1e1e990a2338630776d0554437a00663d4ff195b14e8b 2024-02-14T17:34:35.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 systemd[1]: Started Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:34:35.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.666Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-02-14T17:34:36.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.892Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-02-14T17:34:36.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.946Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-02-14T17:34:36.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.946Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-02-14T17:34:36.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.946Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.476µs wal_replay_duration=478.621029ms total_replay_duration=478.664391ms 2024-02-14T17:34:36.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.950Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:34:36.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.950Z caller=main.go:947 level=info msg="TSDB started" 2024-02-14T17:34:36.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.950Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:34:36.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.970Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.520402ms db_storage=877ns remote_storage=1.516µs web_handler=447ns query_engine=756ns scrape=569.064µs scrape_sd=124.698µs notify=15.945µs notify_sd=19.99µs rules=18.433548ms 2024-02-14T17:34:36.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:34:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:34:35.970Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-14T17:34:36.664 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:36 smithi111 ceph-mon[25603]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:36.664 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:36 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:36.664 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:36 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:36.664 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:36 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:34:36.664 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:36 smithi111 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:34:36.664 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:36 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[24769]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:36.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[24769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[29506]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[29506]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:34:36.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:36 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:38.810 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:38 smithi111 ceph-mon[25603]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:38.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:38 smithi017 ceph-mon[24769]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:38.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:38 smithi017 ceph-mon[29506]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:34:40.866 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:34:40.866 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.867 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:40.868 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:40.868 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.868 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:34:40.868 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:40 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:40.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:34:40.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:40.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:40.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:40.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:34:40.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:40 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:41.809 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:41 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:34:41.809 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:41 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:34:41.810 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:41 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:34:41.810 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:41 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:34:41.810 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:34:41] "GET /metrics HTTP/1.1" 200 37909 "" "Prometheus/2.33.4" 2024-02-14T17:34:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:41 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:34:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:41 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:34:42.710 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:42 smithi017 ceph-mon[29506]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:34:42.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:42 smithi017 ceph-mon[24769]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:34:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:42 smithi111 ceph-mon[25603]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:34:44.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:44 smithi017 ceph-mon[24769]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:44.848 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:44 smithi017 ceph-mon[29506]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:44.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:44 smithi111 ceph-mon[25603]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4256871311' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:45 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4256871311' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:34:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:45 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:45 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:45 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:34:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:45 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:34:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:45 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:34:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:45 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4256871311' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:34:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:46 smithi111 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:34:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:46 smithi111 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:34:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:46 smithi111 ceph-mon[25603]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:46 smithi017 ceph-mon[24769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:34:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:46 smithi017 ceph-mon[24769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:34:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:46 smithi017 ceph-mon[24769]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:47.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:46 smithi017 ceph-mon[29506]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-02-14T17:34:47.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:46 smithi017 ceph-mon[29506]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-02-14T17:34:47.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:46 smithi017 ceph-mon[29506]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:34:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:48 smithi017 ceph-mon[24769]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-14T17:34:48.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:48 smithi017 ceph-mon[29506]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-14T17:34:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:48 smithi111 ceph-mon[25603]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-14T17:34:50.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:50.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:50.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[24769]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[29506]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-14T17:34:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:34:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:34:50.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:34:50.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:50 smithi111 ceph-mon[25603]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-02-14T17:34:50.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:34:52.164 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:34:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:34:51] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-14T17:34:52.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:52 smithi017 ceph-mon[24769]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:52 smithi017 ceph-mon[29506]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:52 smithi111 ceph-mon[25603]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:54 smithi017 ceph-mon[29506]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:54 smithi017 ceph-mon[24769]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:54 smithi111 ceph-mon[25603]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:56 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:56 smithi111 ceph-mon[25603]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:56.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:56 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:56.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:56 smithi017 ceph-mon[24769]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:56.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:56 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:34:56.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:56 smithi017 ceph-mon[29506]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:34:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:34:58 smithi017 ceph-mon[24769]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-14T17:34:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:34:58 smithi017 ceph-mon[29506]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-14T17:34:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:34:58 smithi111 ceph-mon[25603]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-02-14T17:35:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:00 smithi017 ceph-mon[24769]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:00 smithi017 ceph-mon[29506]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:00 smithi111 ceph-mon[25603]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:35:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:35:01] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-14T17:35:02.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:02 smithi017 ceph-mon[24769]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:02.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:02 smithi017 ceph-mon[29506]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:02 smithi111 ceph-mon[25603]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:04 smithi017 ceph-mon[24769]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:04 smithi017 ceph-mon[29506]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:04 smithi111 ceph-mon[25603]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:05 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:05 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:05 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:06 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:06 smithi017 ceph-mon[24769]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:06 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:06 smithi017 ceph-mon[29506]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:06 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:06 smithi111 ceph-mon[25603]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:08 smithi017 ceph-mon[24769]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:08 smithi017 ceph-mon[29506]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:08 smithi111 ceph-mon[25603]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:10 smithi017 ceph-mon[24769]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:10 smithi017 ceph-mon[29506]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:10 smithi111 ceph-mon[25603]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:12.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:35:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:35:11] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-02-14T17:35:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:12 smithi017 ceph-mon[24769]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:12 smithi017 ceph-mon[29506]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:12 smithi111 ceph-mon[25603]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:14 smithi017 ceph-mon[24769]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:14 smithi017 ceph-mon[29506]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:14 smithi111 ceph-mon[25603]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:16 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:16 smithi111 ceph-mon[25603]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:16 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:16 smithi017 ceph-mon[29506]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:16 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:16 smithi017 ceph-mon[24769]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:18 smithi017 ceph-mon[24769]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:18 smithi017 ceph-mon[29506]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:18 smithi111 ceph-mon[25603]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:20 smithi017 ceph-mon[24769]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:20 smithi017 ceph-mon[29506]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:20 smithi111 ceph-mon[25603]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:35:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:35:21] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-02-14T17:35:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:22 smithi017 ceph-mon[24769]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:22 smithi017 ceph-mon[29506]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:22 smithi111 ceph-mon[25603]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:24 smithi017 ceph-mon[24769]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:24 smithi017 ceph-mon[29506]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:24 smithi111 ceph-mon[25603]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:26 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:26 smithi111 ceph-mon[25603]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:26 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:26 smithi017 ceph-mon[24769]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:26 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:26 smithi017 ceph-mon[29506]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:28 smithi017 ceph-mon[24769]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:28 smithi017 ceph-mon[29506]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:28 smithi111 ceph-mon[25603]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:30 smithi017 ceph-mon[24769]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:30 smithi017 ceph-mon[29506]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:30 smithi111 ceph-mon[25603]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:32.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:35:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:35:31] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-02-14T17:35:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:32 smithi017 ceph-mon[24769]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:32 smithi017 ceph-mon[29506]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:32 smithi111 ceph-mon[25603]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:34 smithi017 ceph-mon[24769]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:34 smithi017 ceph-mon[29506]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:34 smithi111 ceph-mon[25603]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:35 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:35 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:35 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:36 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:36 smithi017 ceph-mon[24769]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:36 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:36 smithi017 ceph-mon[29506]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:36 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:36 smithi111 ceph-mon[25603]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:38 smithi017 ceph-mon[24769]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:38.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:38 smithi017 ceph-mon[29506]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:38 smithi111 ceph-mon[25603]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:40 smithi017 ceph-mon[24769]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:40 smithi017 ceph-mon[29506]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:40 smithi111 ceph-mon[25603]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:35:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:35:41] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-02-14T17:35:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:42 smithi017 ceph-mon[24769]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:42 smithi017 ceph-mon[29506]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:42 smithi111 ceph-mon[25603]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:44 smithi017 ceph-mon[24769]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:44 smithi017 ceph-mon[29506]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:44 smithi111 ceph-mon[25603]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:46 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:46.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:46 smithi111 ceph-mon[25603]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:46 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:46 smithi017 ceph-mon[24769]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:46.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:46 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:46.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:46 smithi017 ceph-mon[29506]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:48 smithi017 ceph-mon[24769]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:48 smithi017 ceph-mon[29506]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:48 smithi111 ceph-mon[25603]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:49.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:49 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:35:49.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:49 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:35:49.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:49 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:35:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[24769]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[29506]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:50 smithi111 ceph-mon[25603]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:50.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:35:50.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:50.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:51 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:35:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:51 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:35:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:51 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:51.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:51 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:35:51.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:51 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:35:51.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:51 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:51 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:35:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:51 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:35:51.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:51 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:35:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:35:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:35:51] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-14T17:35:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:52 smithi017 ceph-mon[24769]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:52 smithi017 ceph-mon[29506]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:52 smithi111 ceph-mon[25603]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:54 smithi017 ceph-mon[24769]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:54 smithi017 ceph-mon[29506]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:54 smithi111 ceph-mon[25603]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:56 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:56 smithi111 ceph-mon[25603]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:56 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:56 smithi017 ceph-mon[24769]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:56 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:35:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:56 smithi017 ceph-mon[29506]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:35:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:35:58 smithi017 ceph-mon[24769]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:35:58 smithi017 ceph-mon[29506]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:35:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:35:58 smithi111 ceph-mon[25603]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:00 smithi017 ceph-mon[24769]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:00 smithi017 ceph-mon[29506]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:00 smithi111 ceph-mon[25603]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:36:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:36:01] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-14T17:36:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:02 smithi017 ceph-mon[24769]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:02 smithi017 ceph-mon[29506]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:02 smithi111 ceph-mon[25603]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:04 smithi017 ceph-mon[24769]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:04 smithi017 ceph-mon[29506]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:04 smithi111 ceph-mon[25603]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:05 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:05 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:05.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:05 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:06 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:06 smithi017 ceph-mon[29506]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:06 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:06 smithi017 ceph-mon[24769]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:06 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:06 smithi111 ceph-mon[25603]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:08 smithi017 ceph-mon[24769]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:08 smithi017 ceph-mon[29506]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:08 smithi111 ceph-mon[25603]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:10 smithi017 ceph-mon[24769]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:10 smithi017 ceph-mon[29506]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:10 smithi111 ceph-mon[25603]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:12.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:36:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:36:11] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-02-14T17:36:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:12 smithi017 ceph-mon[24769]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:12 smithi017 ceph-mon[29506]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:12 smithi111 ceph-mon[25603]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:14 smithi017 ceph-mon[24769]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:14 smithi017 ceph-mon[29506]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:14 smithi111 ceph-mon[25603]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:16 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:16.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:16 smithi111 ceph-mon[25603]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:16 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:16 smithi017 ceph-mon[29506]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:16 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:16 smithi017 ceph-mon[24769]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:18 smithi017 ceph-mon[29506]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:18 smithi017 ceph-mon[24769]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:18 smithi111 ceph-mon[25603]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:20 smithi017 ceph-mon[24769]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:20 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:20 smithi017 ceph-mon[29506]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:20.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:20 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:20 smithi111 ceph-mon[25603]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:20 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:36:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:36:21] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-14T17:36:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:22 smithi017 ceph-mon[24769]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:22 smithi017 ceph-mon[29506]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:22 smithi111 ceph-mon[25603]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:24 smithi017 ceph-mon[24769]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:24 smithi017 ceph-mon[29506]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:24.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:24 smithi111 ceph-mon[25603]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:26 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:26.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:26 smithi111 ceph-mon[25603]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:26 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:26 smithi017 ceph-mon[24769]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:26 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:26 smithi017 ceph-mon[29506]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:28 smithi017 ceph-mon[24769]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:28 smithi017 ceph-mon[29506]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:28 smithi111 ceph-mon[25603]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:30 smithi017 ceph-mon[29506]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:30 smithi017 ceph-mon[24769]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:30 smithi111 ceph-mon[25603]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:32.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:36:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:36:31] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-14T17:36:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:32 smithi017 ceph-mon[24769]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:32 smithi017 ceph-mon[29506]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:32 smithi111 ceph-mon[25603]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:34 smithi017 ceph-mon[29506]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:34 smithi017 ceph-mon[24769]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:34 smithi111 ceph-mon[25603]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:35 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:35 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:35 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:36 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:36 smithi017 ceph-mon[24769]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:36 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:36 smithi017 ceph-mon[29506]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:36 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:36 smithi111 ceph-mon[25603]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:38.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:38 smithi017 ceph-mon[29506]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:38 smithi017 ceph-mon[24769]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:38 smithi111 ceph-mon[25603]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:40 smithi017 ceph-mon[24769]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:40 smithi017 ceph-mon[29506]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:40 smithi111 ceph-mon[25603]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:36:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:36:41] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-02-14T17:36:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:42 smithi017 ceph-mon[24769]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:42 smithi017 ceph-mon[29506]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:42 smithi111 ceph-mon[25603]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:44 smithi017 ceph-mon[24769]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:44.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:44 smithi017 ceph-mon[29506]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:44 smithi111 ceph-mon[25603]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:46.536 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:46 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:46.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:46 smithi111 ceph-mon[25603]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:46 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:46 smithi017 ceph-mon[24769]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:46 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:46 smithi017 ceph-mon[29506]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:48 smithi017 ceph-mon[24769]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:48 smithi017 ceph-mon[29506]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:48 smithi111 ceph-mon[25603]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:50 smithi017 ceph-mon[24769]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:50 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:50 smithi017 ceph-mon[29506]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:50 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:50 smithi111 ceph-mon[25603]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:50 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:36:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:36:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:36:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:36:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:36:51.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:36:51.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:36:51.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:36:51.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:36:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:51 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:36:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:51 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:36:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:51 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:36:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:51 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' 2024-02-14T17:36:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:36:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:36:51] "GET /metrics HTTP/1.1" 200 37912 "" "Prometheus/2.33.4" 2024-02-14T17:36:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:52 smithi017 ceph-mon[24769]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:52 smithi017 ceph-mon[29506]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:52 smithi111 ceph-mon[25603]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:54 smithi017 ceph-mon[24769]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:54 smithi017 ceph-mon[29506]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:54 smithi111 ceph-mon[25603]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:56 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:56 smithi017 ceph-mon[24769]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:56 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:56 smithi017 ceph-mon[29506]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:56 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:36:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:56 smithi111 ceph-mon[25603]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:36:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:36:58 smithi017 ceph-mon[24769]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:36:58 smithi017 ceph-mon[29506]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:36:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:36:58 smithi111 ceph-mon[25603]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:00 smithi017 ceph-mon[24769]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:00 smithi017 ceph-mon[29506]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:00 smithi111 ceph-mon[25603]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:37:01] "GET /metrics HTTP/1.1" 200 37912 "" "Prometheus/2.33.4" 2024-02-14T17:37:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:02 smithi017 ceph-mon[24769]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:02 smithi017 ceph-mon[29506]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:02 smithi111 ceph-mon[25603]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:04 smithi017 ceph-mon[24769]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:04 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:04 smithi017 ceph-mon[29506]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:04 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:04 smithi111 ceph-mon[25603]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:04 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:06 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:06 smithi017 ceph-mon[24769]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:06 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:06 smithi017 ceph-mon[29506]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:06 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:06 smithi111 ceph-mon[25603]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:08 smithi017 ceph-mon[24769]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:08 smithi017 ceph-mon[29506]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:08 smithi111 ceph-mon[25603]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:10 smithi017 ceph-mon[24769]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:10 smithi017 ceph-mon[29506]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:10 smithi111 ceph-mon[25603]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:12.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:37:11] "GET /metrics HTTP/1.1" 200 37912 "" "Prometheus/2.33.4" 2024-02-14T17:37:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:12 smithi017 ceph-mon[24769]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:12 smithi017 ceph-mon[29506]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:12 smithi111 ceph-mon[25603]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:14 smithi017 ceph-mon[29506]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:14 smithi017 ceph-mon[24769]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:14 smithi111 ceph-mon[25603]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:16.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:16 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:16.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:16 smithi111 ceph-mon[25603]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:16 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:16 smithi017 ceph-mon[24769]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:16 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:16.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:16 smithi017 ceph-mon[29506]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:18 smithi017 ceph-mon[24769]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:18 smithi017 ceph-mon[29506]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:18 smithi111 ceph-mon[25603]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:19.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:19 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:19 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:19.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:19 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:20 smithi017 ceph-mon[24769]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:20 smithi017 ceph-mon[29506]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:20 smithi111 ceph-mon[25603]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:21.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:37:21] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.33.4" 2024-02-14T17:37:22.435 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:37:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:22 smithi017 ceph-mon[24769]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:22 smithi017 ceph-mon[29506]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:22 smithi111 ceph-mon[25603]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (9m) 2m ago 17m 24.4M - 0.23.0 ba2b418f427c 556f84c8c0ad 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (8m) 2m ago 16m 55.1M - 8.3.5 dad864ee21e9 3af96da5f3f2 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (2m) 2m ago 16m 47.9M - 3.5 e1d6a67b021e 8ecd81b68f43 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283 running (11m) 2m ago 21m 447M - 19.0.0-1430-g8727ce93 170846cf6432 2ffaf8e8fceb 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (6m) 2m ago 22m 496M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (22m) 2m ago 23m 60.5M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (21m) 2m ago 21m 47.5M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (21m) 2m ago 21m 44.6M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (9m) 2m ago 17m 17.3M - 1.3.1 1dbe0e931976 f9e29bdc31de 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (8m) 2m ago 17m 17.1M - 1.3.1 1dbe0e931976 5d05f9239e5a 2024-02-14T17:37:24.423 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (21m) 2m ago 21m 53.8M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (20m) 2m ago 20m 53.0M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (20m) 2m ago 20m 51.6M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (19m) 2m ago 19m 50.4M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (19m) 2m ago 19m 52.4M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (19m) 2m ago 19m 54.0M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (18m) 2m ago 18m 49.5M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (18m) 2m ago 18m 54.9M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (2m) 2m ago 17m 46.6M - 2.33.4 514e6a882f6e 877352eda179 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (16m) 2m ago 16m 75.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:37:24.424 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (16m) 2m ago 16m 76.0M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:37:24.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:24 smithi017 ceph-mon[24769]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:24 smithi017 ceph-mon[29506]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:24 smithi111 ceph-mon[25603]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:25.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions' 2024-02-14T17:37:25.618 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:25 smithi017 ceph-mon[24769]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:25.618 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:25 smithi017 ceph-mon[29506]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:25.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:25 smithi111 ceph-mon[25603]: from='client.25180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:37:26.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:26 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:26.642 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:26 smithi017 ceph-mon[24769]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:26.642 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:26 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:26.642 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:26 smithi017 ceph-mon[29506]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:26 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:26 smithi111 ceph-mon[25603]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:37:27.123 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:37:27.124 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:37:27.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:27 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/998447190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:37:27.688 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:27 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/998447190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:37:27.712 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph -s' 2024-02-14T17:37:27.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:27 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/998447190' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:37:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:28 smithi017 ceph-mon[24769]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:28 smithi017 ceph-mon[29506]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:28 smithi111 ceph-mon[25603]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: cluster: 2024-02-14T17:37:29.748 INFO:teuthology.orchestra.run.smithi017.stdout: id: 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: health: HEALTH_OK 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: services: 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: mgr: y(active, since 3m), standbys: x 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: data: 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: pools: 6 pools, 161 pgs 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: objects: 209 objects, 585 KiB 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: usage: 94 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: pgs: 161 active+clean 2024-02-14T17:37:29.749 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:37:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: io: 2024-02-14T17:37:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-02-14T17:37:29.750 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-02-14T17:37:30.331 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:37:30.641 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:30 smithi017 ceph-mon[24769]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:30.641 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:30 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2926674028' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:37:30.641 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:30 smithi017 ceph-mon[29506]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:30.641 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:30 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2926674028' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:37:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:30 smithi111 ceph-mon[25603]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:30.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:30 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2926674028' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-14T17:37:31.884 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:37:31] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.33.4" 2024-02-14T17:37:32.382 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_OK 2024-02-14T17:37:32.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:32 smithi017 ceph-mon[24769]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:32.676 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:32 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/141265034' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:37:32.676 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:32 smithi017 ceph-mon[29506]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:32.676 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:32 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/141265034' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:37:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:32 smithi111 ceph-mon[25603]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:32 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/141265034' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:37:32.972 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-14T17:37:34.575 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:34 smithi017 ceph-mon[24769]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:34.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:34 smithi017 ceph-mon[24769]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:34.576 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:34 smithi017 ceph-mon[29506]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:34.576 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:34 smithi017 ceph-mon[29506]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:34 smithi111 ceph-mon[25603]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:34.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:34 smithi111 ceph-mon[25603]: from='mgr.25090 172.21.15.17:0/3054735575' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:35.104 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:37:35.660 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:35 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1490602841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:37:35.660 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:35 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1490602841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:37:35.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph mgr fail' 2024-02-14T17:37:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:35 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1490602841' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:37:36.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:36 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:36.678 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:36 smithi017 ceph-mon[24769]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:36.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:36 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:36.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:36 smithi017 ceph-mon[29506]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:36 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:36 smithi111 ceph-mon[25603]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:37:38.690 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-mon[25603]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:38.690 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/616564707' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:37:38.690 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-mon[25603]: osdmap e103: 8 total, 8 up, 8 in 2024-02-14T17:37:38.690 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:37:38] ENGINE Bus STOPPING 2024-02-14T17:37:38.691 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:37:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:37:38.691 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:37:38] ENGINE Bus STOPPED 2024-02-14T17:37:38.691 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:37:38] ENGINE Bus STARTING 2024-02-14T17:37:38.691 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:37:38] ENGINE Serving on http://:::9283 2024-02-14T17:37:38.691 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:37:38] ENGINE Bus STARTED 2024-02-14T17:37:38.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-mon[29506]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:38.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/616564707' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:37:38.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-mon[29506]: osdmap e103: 8 total, 8 up, 8 in 2024-02-14T17:37:38.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-mon[24769]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:38.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/616564707' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-02-14T17:37:38.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-mon[24769]: osdmap e103: 8 total, 8 up, 8 in 2024-02-14T17:37:38.693 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:38.446+0000 7f57407c7700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-14T17:37:38.693 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ignoring --setuser ceph since I am not root 2024-02-14T17:37:38.693 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ignoring --setgroup ceph since I am not root 2024-02-14T17:37:38.693 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:38 smithi017 ceph-mgr[62961]: -- 172.21.15.17:0/3758221293 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f5fe8bc1a0 con 0x55f5fe892c00 2024-02-14T17:37:39.103 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'sleep 180' 2024-02-14T17:37:39.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:39.047+0000 7f1213fa0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:37:39.425 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:39.163+0000 7f1213fa0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:37:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/616564707' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: mgrmap e32: x(active, starting, since 0.712854s) 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:37:39.708 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: Manager daemon x is now available 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:39.709 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: [14/Feb/2024:17:37:38] ENGINE Bus STARTING 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: [14/Feb/2024:17:37:38] ENGINE Serving on http://172.21.15.111:8765 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: [14/Feb/2024:17:37:39] ENGINE Serving on https://172.21.15.111:7150 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[29506]: [14/Feb/2024:17:37:39] ENGINE Bus STARTED 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/616564707' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: mgrmap e32: x(active, starting, since 0.712854s) 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:37:39.710 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: Manager daemon x is now available 2024-02-14T17:37:39.711 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: [14/Feb/2024:17:37:38] ENGINE Bus STARTING 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: [14/Feb/2024:17:37:38] ENGINE Serving on http://172.21.15.111:8765 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: [14/Feb/2024:17:37:39] ENGINE Serving on https://172.21.15.111:7150 2024-02-14T17:37:39.712 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-mon[24769]: [14/Feb/2024:17:37:39] ENGINE Bus STARTED 2024-02-14T17:37:39.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/616564707' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-02-14T17:37:39.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: mgrmap e32: x(active, starting, since 0.712854s) 2024-02-14T17:37:39.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:37:39.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:37:39.719 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: Manager daemon x is now available 2024-02-14T17:37:39.720 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: [14/Feb/2024:17:37:38] ENGINE Bus STARTING 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: [14/Feb/2024:17:37:38] ENGINE Serving on http://172.21.15.111:8765 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: [14/Feb/2024:17:37:39] ENGINE Serving on https://172.21.15.111:7150 2024-02-14T17:37:39.721 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:39 smithi111 ceph-mon[25603]: [14/Feb/2024:17:37:39] ENGINE Bus STARTED 2024-02-14T17:37:40.127 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:39.867+0000 7f1213fa0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:37:40.127 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:39 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:39.963+0000 7f1213fa0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:37:40.127 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:40.124+0000 7f1213fa0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:37:40.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:40.224+0000 7f1213fa0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:37:40.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:40.311+0000 7f1213fa0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:37:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:40 smithi111 ceph-mon[25603]: mgrmap e33: x(active, since 1.71893s) 2024-02-14T17:37:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:40 smithi111 ceph-mon[25603]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:40.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-mon[24769]: mgrmap e33: x(active, since 1.71893s) 2024-02-14T17:37:40.704 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-mon[24769]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:40.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-mon[29506]: mgrmap e33: x(active, since 1.71893s) 2024-02-14T17:37:40.705 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-mon[29506]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:40.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:40.588+0000 7f1213fa0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:37:40.705 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:40.702+0000 7f1213fa0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:37:40.994 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:40.795+0000 7f1213fa0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:37:40.995 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:40 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:40.894+0000 7f1213fa0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:37:41.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:41.077+0000 7f1213fa0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:37:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:41 smithi111 ceph-mon[25603]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:41.736 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:41 smithi017 ceph-mon[24769]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:41.737 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:41 smithi017 ceph-mon[29506]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:42.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:42.112+0000 7f1213fa0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:37:42.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:42.248+0000 7f1213fa0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:37:42.718 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:42 smithi111 ceph-mon[25603]: mgrmap e34: x(active, since 3s) 2024-02-14T17:37:42.740 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:42 smithi017 ceph-mon[24769]: mgrmap e34: x(active, since 3s) 2024-02-14T17:37:42.740 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:42 smithi017 ceph-mon[29506]: mgrmap e34: x(active, since 3s) 2024-02-14T17:37:42.740 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:42.627+0000 7f1213fa0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:37:42.740 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:42.738+0000 7f1213fa0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:37:43.052 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:42 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:42.945+0000 7f1213fa0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:37:43.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:43.143+0000 7f1213fa0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:37:43.727 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:43.454+0000 7f1213fa0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:37:43.727 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:43.724+0000 7f1213fa0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:37:43.981 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:43.972+0000 7f1213fa0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:37:44.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:43 smithi111 ceph-mon[25603]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:44.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:43 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:44.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:43 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:44.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-mon[24769]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:44.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:44.308 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:44.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-mon[29506]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:37:44.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:44.309 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:43 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:44.851+0000 7f1213fa0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:37:45.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:45 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:45 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:45 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:45 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:45.805 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:45.460+0000 7f1213fa0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:37:45.805 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:45.570+0000 7f1213fa0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:37:45.805 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:45.803+0000 7f1213fa0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:37:46.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:37:45] ENGINE Bus STARTING 2024-02-14T17:37:46.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: CherryPy Checker: 2024-02-14T17:37:46.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: The Application mounted at '' has an empty config. 2024-02-14T17:37:46.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:37:46.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:37:45] ENGINE Serving on http://:::9283 2024-02-14T17:37:46.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:45 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:37:45] ENGINE Bus STARTED 2024-02-14T17:37:46.667 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: Standby manager daemon y started 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:37:46.668 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:46 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:37:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-14T17:37:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: Standby manager daemon y started 2024-02-14T17:37:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:37:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='client.24986 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: Standby manager daemon y started 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-14T17:37:46.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:37:46.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:46.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:46.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:37:46.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:46 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:37:47.592 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:37:47.592 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:37:47.592 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: mgrmap e35: x(active, since 8s), standbys: y 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:37:47.593 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:47 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:37:47.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:37:47.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: mgrmap e35: x(active, since 8s), standbys: y 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:37:47.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: mgrmap e35: x(active, since 8s), standbys: y 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:37:47.616 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:47 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:37:48.560 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:37:48.560 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:37:48.560 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[29506]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:37:48.560 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:37:48.560 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:37:48.560 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[24769]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:37:48.561 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:48 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:37:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:48 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:37:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:48 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:37:48.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:48 smithi111 ceph-mon[25603]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:37:48.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:48 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:37:48.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:48 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:37:48.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:48 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:37:48.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:48 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:37:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-mon[25603]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:37:50.112+0000 7fe2d57c2700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:50.485 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:50.486 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return future.result(timeout) 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.__get_result() 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: raise self._exception 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:37:50.487 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: config_blobs=daemon_spec.final_config, 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:50.488 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:50.489 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:37:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:50 smithi017 ceph-mon[24769]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:37:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:50 smithi017 ceph-mon[29506]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:51 smithi111 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:37:51.485 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:37:51.486 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: return future.result(timeout) 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.__get_result() 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: raise self._exception 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:51.487 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:51.488 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:51.489 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:51.489 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:51.489 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:51.489 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.489 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:51 smithi111 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:37:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:51 smithi017 ceph-mon[24769]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:37:51.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:37:51.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:51.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:51.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:51 smithi017 ceph-mon[24769]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:51 smithi017 ceph-mon[29506]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:51.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-02-14T17:37:51.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:37:51.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:37:51.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:51 smithi017 ceph-mon[29506]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:37:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:37:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:37:51] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-02-14T17:37:52.534 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:52 smithi111 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:37:52.534 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:52 smithi111 ceph-mon[25603]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:37:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:52 smithi017 ceph-mon[24769]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:37:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:52 smithi017 ceph-mon[24769]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:37:52.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:52 smithi017 ceph-mon[29506]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:37:52.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:52 smithi017 ceph-mon[29506]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-14T17:37:53.568 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 systemd[1]: Stopping Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:37:53.568 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 bash[57733]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:37:53.967 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:37:53.967 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:37:53.967 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:37:53.967 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-14T17:37:53.967 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:37:53.967 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-14T17:37:53.967 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.566Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[54774]: ts=2024-02-14T17:37:53.567Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 podman[57745]: 2024-02-14 17:37:53.576362876 +0000 UTC m=+0.127448079 container died 877352eda179a0572ce1e1e990a2338630776d0554437a00663d4ff195b14e8b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 podman[57745]: 2024-02-14 17:37:53.965699844 +0000 UTC m=+0.516785058 container cleanup 877352eda179a0572ce1e1e990a2338630776d0554437a00663d4ff195b14e8b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:37:53.968 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 bash[57745]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:37:54.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:53 smithi111 bash[57773]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:37:54.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 podman[57759]: 2024-02-14 17:37:54.183621076 +0000 UTC m=+0.609139064 container remove 877352eda179a0572ce1e1e990a2338630776d0554437a00663d4ff195b14e8b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:37:54.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-14T17:37:54.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 systemd[1]: Stopped Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:37:54.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Consumed 1.293s CPU time. 2024-02-14T17:37:54.235 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 systemd[1]: Starting Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:37:54.574 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-mon[25603]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:37:54.574 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:54.575 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 podman[57842]: 2024-02-14 17:37:54.317135857 +0000 UTC m=+0.018604761 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-14T17:37:54.575 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 podman[57842]: 2024-02-14 17:37:54.451355824 +0000 UTC m=+0.152824726 container create 8006aec214efb08dabfb8565436bafa38a4aea403624d439504f85d1f60bc973 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:37:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:54 smithi017 ceph-mon[24769]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:37:54.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:54 smithi017 ceph-mon[29506]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:37:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:37:54.947 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 podman[57842]: 2024-02-14 17:37:54.704047404 +0000 UTC m=+0.405516314 container init 8006aec214efb08dabfb8565436bafa38a4aea403624d439504f85d1f60bc973 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:37:54.947 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 podman[57842]: 2024-02-14 17:37:54.706620697 +0000 UTC m=+0.408089608 container start 8006aec214efb08dabfb8565436bafa38a4aea403624d439504f85d1f60bc973 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.729Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.729Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.729Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.729Z caller=main.go:518 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi111 (none))" 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.729Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.729Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.731Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.731Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.733Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.734Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.734Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.682µs 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.734Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:37:54.948 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 bash[57842]: 8006aec214efb08dabfb8565436bafa38a4aea403624d439504f85d1f60bc973 2024-02-14T17:37:54.949 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 systemd[1]: Started Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:37:54.949 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:54.946Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-02-14T17:37:55.220 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.182Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-02-14T17:37:55.220 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.219Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.273Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.274Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.274Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.954µs wal_replay_duration=539.17156ms total_replay_duration=539.220866ms 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.277Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.277Z caller=main.go:947 level=info msg="TSDB started" 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.277Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.296Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.042234ms db_storage=881ns remote_storage=1.196µs web_handler=345ns query_engine=713ns scrape=402.286µs scrape_sd=85.1µs notify=10.596µs notify_sd=14.83µs rules=18.217987ms 2024-02-14T17:37:55.485 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:37:55.296Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-14T17:37:55.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-mon[25603]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:37:55.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:55.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:55.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:37:55.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-mon[25603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:37:55.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:55 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[29506]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:37:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:37:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[29506]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:37:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[24769]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-14T17:37:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:37:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:37:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[24769]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:37:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:55 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:37:58.158 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:37:57 smithi111 ceph-mon[25603]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:37:57 smithi017 ceph-mon[24769]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:37:58.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:37:57 smithi017 ceph-mon[29506]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:00.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:00 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:38:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:38:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:38:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:38:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:38:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:38:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:38:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:38:00.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:00 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:38:02.279 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:02 smithi017 ceph-mon[24769]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:38:02.280 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:02 smithi017 ceph-mon[29506]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:38:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-mon[25603]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:38:02.738+0000 7fe2d57c2700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:03.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:03.236 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:38:03.237 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return future.result(timeout) 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.__get_result() 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: raise self._exception 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: config_blobs=daemon_spec.final_config, 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:03.238 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:03.239 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:03.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:03.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:03.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:03.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:03.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:03.240 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:03.512 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:03 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/4152250487' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:38:03.512 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:03 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1793116539' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/215058162"}]: dispatch 2024-02-14T17:38:03.512 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:03 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/4152250487' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:38:03.513 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:03 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1793116539' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/215058162"}]: dispatch 2024-02-14T17:38:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:03 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/4152250487' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:38:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:03 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1793116539' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/215058162"}]: dispatch 2024-02-14T17:38:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[29506]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:38:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[29506]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:38:04.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:04.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:38:04.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:04.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:04.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[24769]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:38:04.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[24769]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:04.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:38:04.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1793116539' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/215058162"}]': finished 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[29506]: osdmap e104: 8 total, 8 up, 8 in 2024-02-14T17:38:04.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/72358383' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/884602878"}]: dispatch 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.700 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1793116539' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/215058162"}]': finished 2024-02-14T17:38:04.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[24769]: osdmap e104: 8 total, 8 up, 8 in 2024-02-14T17:38:04.701 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:04 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/72358383' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/884602878"}]: dispatch 2024-02-14T17:38:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:04 smithi111 ceph-mon[25603]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:38:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:04 smithi111 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:38:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:38:04.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: return future.result(timeout) 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.__get_result() 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: raise self._exception 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi-foo-smithi017-wtnmjz 2024-02-14T17:38:04.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: /usr/bin/podman: stderr Error: no such container ceph-612435b4-cb5c-11ee-95b9-87774f69a715-iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:38:04.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:04 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1793116539' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/215058162"}]': finished 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:04 smithi111 ceph-mon[25603]: osdmap e104: 8 total, 8 up, 8 in 2024-02-14T17:38:04.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:04 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/72358383' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/884602878"}]: dispatch 2024-02-14T17:38:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:05 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/72358383' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/884602878"}]': finished 2024-02-14T17:38:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:05 smithi017 ceph-mon[24769]: osdmap e105: 8 total, 8 up, 8 in 2024-02-14T17:38:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:05 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/999362667' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/252905691"}]: dispatch 2024-02-14T17:38:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:05 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/72358383' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/884602878"}]': finished 2024-02-14T17:38:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:05 smithi017 ceph-mon[29506]: osdmap e105: 8 total, 8 up, 8 in 2024-02-14T17:38:05.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:05 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/999362667' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/252905691"}]: dispatch 2024-02-14T17:38:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:05 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/72358383' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/884602878"}]': finished 2024-02-14T17:38:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:05 smithi111 ceph-mon[25603]: osdmap e105: 8 total, 8 up, 8 in 2024-02-14T17:38:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:05 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/999362667' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/252905691"}]: dispatch 2024-02-14T17:38:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[24769]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-14T17:38:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/999362667' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/252905691"}]': finished 2024-02-14T17:38:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[24769]: osdmap e106: 8 total, 8 up, 8 in 2024-02-14T17:38:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1453224677' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3151870141"}]: dispatch 2024-02-14T17:38:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[29506]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-14T17:38:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/999362667' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/252905691"}]': finished 2024-02-14T17:38:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[29506]: osdmap e106: 8 total, 8 up, 8 in 2024-02-14T17:38:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:06 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1453224677' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3151870141"}]: dispatch 2024-02-14T17:38:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:06 smithi111 ceph-mon[25603]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-02-14T17:38:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:06 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/999362667' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/252905691"}]': finished 2024-02-14T17:38:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:06 smithi111 ceph-mon[25603]: osdmap e106: 8 total, 8 up, 8 in 2024-02-14T17:38:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:06 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1453224677' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3151870141"}]: dispatch 2024-02-14T17:38:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:07 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/1453224677' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3151870141"}]': finished 2024-02-14T17:38:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:07 smithi111 ceph-mon[25603]: osdmap e107: 8 total, 8 up, 8 in 2024-02-14T17:38:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:07 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2193126924' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]: dispatch 2024-02-14T17:38:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:07 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]: dispatch 2024-02-14T17:38:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/1453224677' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3151870141"}]': finished 2024-02-14T17:38:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[24769]: osdmap e107: 8 total, 8 up, 8 in 2024-02-14T17:38:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2193126924' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]: dispatch 2024-02-14T17:38:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]: dispatch 2024-02-14T17:38:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/1453224677' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3151870141"}]': finished 2024-02-14T17:38:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[29506]: osdmap e107: 8 total, 8 up, 8 in 2024-02-14T17:38:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2193126924' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]: dispatch 2024-02-14T17:38:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:07 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]: dispatch 2024-02-14T17:38:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:38:07] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-02-14T17:38:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[24769]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:38:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]': finished 2024-02-14T17:38:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[24769]: osdmap e108: 8 total, 8 up, 8 in 2024-02-14T17:38:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/973594639' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]: dispatch 2024-02-14T17:38:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]: dispatch 2024-02-14T17:38:08.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[29506]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:38:08.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]': finished 2024-02-14T17:38:08.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[29506]: osdmap e108: 8 total, 8 up, 8 in 2024-02-14T17:38:08.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/973594639' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]: dispatch 2024-02-14T17:38:08.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:08 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]: dispatch 2024-02-14T17:38:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:08 smithi111 ceph-mon[25603]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:38:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:08 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2979608737"}]': finished 2024-02-14T17:38:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:08 smithi111 ceph-mon[25603]: osdmap e108: 8 total, 8 up, 8 in 2024-02-14T17:38:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:08 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/973594639' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]: dispatch 2024-02-14T17:38:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:08 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]: dispatch 2024-02-14T17:38:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:09 smithi017 ceph-mon[29506]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]': finished 2024-02-14T17:38:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:09 smithi017 ceph-mon[29506]: osdmap e109: 8 total, 8 up, 8 in 2024-02-14T17:38:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:09 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:09.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:09 smithi017 ceph-mon[24769]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]': finished 2024-02-14T17:38:09.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:09 smithi017 ceph-mon[24769]: osdmap e109: 8 total, 8 up, 8 in 2024-02-14T17:38:09.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:09 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:09.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:09 smithi111 ceph-mon[25603]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2979608737"}]': finished 2024-02-14T17:38:09.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:09 smithi111 ceph-mon[25603]: osdmap e109: 8 total, 8 up, 8 in 2024-02-14T17:38:09.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:09 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:10 smithi017 ceph-mon[24769]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:38:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:10 smithi017 ceph-mon[29506]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:38:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:10 smithi111 ceph-mon[25603]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:38:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:12 smithi017 ceph-mon[29506]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-02-14T17:38:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:12 smithi017 ceph-mon[24769]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-02-14T17:38:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:12 smithi111 ceph-mon[25603]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-02-14T17:38:13.582 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:13 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:13.582 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:13 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:13 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:14 smithi017 ceph-mon[24769]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 826 B/s rd, 0 op/s 2024-02-14T17:38:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:14 smithi017 ceph-mon[29506]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 826 B/s rd, 0 op/s 2024-02-14T17:38:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:14 smithi111 ceph-mon[25603]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 826 B/s rd, 0 op/s 2024-02-14T17:38:16.561 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:16 smithi111 ceph-mon[25603]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:16 smithi017 ceph-mon[24769]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:16 smithi017 ceph-mon[29506]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:38:17] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-02-14T17:38:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:18 smithi017 ceph-mon[24769]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:38:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:18 smithi017 ceph-mon[29506]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:38:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:18 smithi111 ceph-mon[25603]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:38:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:20 smithi017 ceph-mon[24769]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-02-14T17:38:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:20 smithi017 ceph-mon[29506]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-02-14T17:38:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:20 smithi111 ceph-mon[25603]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-02-14T17:38:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:22 smithi017 ceph-mon[24769]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:22 smithi017 ceph-mon[29506]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:22 smithi111 ceph-mon[25603]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:23.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:23 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:23.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:23 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:23 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:24 smithi017 ceph-mon[29506]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:24 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:24 smithi017 ceph-mon[24769]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:24 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:24 smithi111 ceph-mon[25603]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:24 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:26 smithi017 ceph-mon[29506]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:26 smithi017 ceph-mon[24769]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:26 smithi111 ceph-mon[25603]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:38:27] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-02-14T17:38:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:28 smithi017 ceph-mon[24769]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:28 smithi017 ceph-mon[29506]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:28 smithi111 ceph-mon[25603]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:30 smithi017 ceph-mon[24769]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:30 smithi017 ceph-mon[29506]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:30 smithi111 ceph-mon[25603]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:32.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:32 smithi017 ceph-mon[24769]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:32 smithi017 ceph-mon[29506]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:32 smithi111 ceph-mon[25603]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:33.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:33 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:33.677 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:33 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:33 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:34 smithi017 ceph-mon[24769]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:34 smithi017 ceph-mon[29506]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:34 smithi111 ceph-mon[25603]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:36 smithi017 ceph-mon[24769]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:36 smithi017 ceph-mon[29506]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:36 smithi111 ceph-mon[25603]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:37 smithi017 ceph-mon[24769]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:37.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:37 smithi017 ceph-mon[29506]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:37 smithi111 ceph-mon[25603]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:37.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:38:37] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-02-14T17:38:39.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:39 smithi017 ceph-mon[24769]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:39.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:39 smithi017 ceph-mon[29506]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:39.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:39 smithi111 ceph-mon[25603]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:39.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:42 smithi017 ceph-mon[24769]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:42 smithi017 ceph-mon[29506]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:42 smithi111 ceph-mon[25603]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:43.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:43 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:43.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:43 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:43 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:44 smithi017 ceph-mon[29506]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:44 smithi017 ceph-mon[24769]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:44 smithi111 ceph-mon[25603]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:46.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:46 smithi111 ceph-mon[25603]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:46 smithi017 ceph-mon[24769]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:46.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:46 smithi017 ceph-mon[29506]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:38:47] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-02-14T17:38:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:48 smithi017 ceph-mon[24769]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:48 smithi017 ceph-mon[29506]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:48 smithi111 ceph-mon[25603]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:50 smithi017 ceph-mon[24769]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:50 smithi017 ceph-mon[29506]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:50 smithi111 ceph-mon[25603]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:52 smithi017 ceph-mon[24769]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:52 smithi017 ceph-mon[29506]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:52 smithi111 ceph-mon[25603]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:53 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:53 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:53 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:38:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:54 smithi017 ceph-mon[24769]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:54 smithi017 ceph-mon[29506]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:54 smithi111 ceph-mon[25603]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:38:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:56 smithi017 ceph-mon[24769]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:56 smithi017 ceph-mon[29506]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:56 smithi111 ceph-mon[25603]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:38:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:38:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:38:57] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-14T17:38:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:38:58 smithi017 ceph-mon[24769]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:38:58 smithi017 ceph-mon[29506]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:38:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:38:58 smithi111 ceph-mon[25603]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:00 smithi017 ceph-mon[24769]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:00 smithi017 ceph-mon[29506]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:00 smithi111 ceph-mon[25603]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:02 smithi017 ceph-mon[24769]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:02 smithi017 ceph-mon[29506]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:02 smithi111 ceph-mon[25603]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:03 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:03 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:03.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:03 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:03.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:03 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:03 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:03 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:39:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[24769]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[29506]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:39:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:04 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:04 smithi111 ceph-mon[25603]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:04 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:04 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:39:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:04 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:39:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:04 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:39:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:04 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:39:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:04 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:39:05.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:05 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:39:05.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:05 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:39:05.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:05 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:39:05.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:05 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:39:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:39:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:39:05.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:39:05.357+0000 7fe2d57c2700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:05.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:39:05.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:05.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:05.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:05.736 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return future.result(timeout) 2024-02-14T17:39:05.737 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.__get_result() 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: raise self._exception 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: config_blobs=daemon_spec.final_config, 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:05.738 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:05.739 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:05.739 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:39:05.739 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:05.739 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:39:05.739 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:05.740 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:06 smithi017 ceph-mon[24769]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:06 smithi017 ceph-mon[29506]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:06 smithi111 ceph-mon[25603]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:07 smithi111 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:39:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.608 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: return future.result(timeout) 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.__get_result() 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: raise self._exception 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:39:07.609 INFO:journalctl@ceph.mon.b.smithi111.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:07.610 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.611 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:07 smithi017 ceph-mon[24769]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:39:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:39:07.692 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:39:07.693 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:07.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:07 smithi017 ceph-mon[29506]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.695 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.696 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:39:07.697 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:39:07.698 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:39:07.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:39:07.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:39:07] "GET /metrics HTTP/1.1" 200 37905 "" "Prometheus/2.33.4" 2024-02-14T17:39:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:08 smithi017 ceph-mon[24769]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:08 smithi017 ceph-mon[29506]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:08 smithi111 ceph-mon[25603]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:09.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:09 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:09 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:09.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:09 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:10 smithi017 ceph-mon[24769]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:10.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:10 smithi017 ceph-mon[29506]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:10 smithi111 ceph-mon[25603]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:12 smithi017 ceph-mon[24769]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:12 smithi017 ceph-mon[29506]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:12 smithi111 ceph-mon[25603]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:13.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:13 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:13 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:13 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:14 smithi017 ceph-mon[24769]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:14 smithi017 ceph-mon[29506]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:14 smithi111 ceph-mon[25603]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:16 smithi017 ceph-mon[24769]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:16 smithi017 ceph-mon[29506]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:16 smithi111 ceph-mon[25603]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:39:17] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-02-14T17:39:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:18 smithi017 ceph-mon[24769]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:18 smithi017 ceph-mon[29506]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:18 smithi111 ceph-mon[25603]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:20 smithi017 ceph-mon[24769]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:20 smithi017 ceph-mon[29506]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:20 smithi111 ceph-mon[25603]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:22 smithi017 ceph-mon[24769]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:22 smithi017 ceph-mon[29506]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:22 smithi111 ceph-mon[25603]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:24 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:24 smithi017 ceph-mon[24769]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:24 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:24 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:24 smithi017 ceph-mon[29506]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:24.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:24 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:24 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:24 smithi111 ceph-mon[25603]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:24 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:26 smithi017 ceph-mon[24769]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:26 smithi017 ceph-mon[29506]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:26 smithi111 ceph-mon[25603]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:39:27] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-02-14T17:39:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:28 smithi017 ceph-mon[24769]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:28 smithi017 ceph-mon[29506]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:28 smithi111 ceph-mon[25603]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:30 smithi017 ceph-mon[24769]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:30 smithi017 ceph-mon[29506]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:30 smithi111 ceph-mon[25603]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:31 smithi017 ceph-mon[24769]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:31.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:31 smithi017 ceph-mon[29506]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:31 smithi111 ceph-mon[25603]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:33.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:33 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:33.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:33 smithi017 ceph-mon[24769]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:33.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:33 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:33.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:33 smithi017 ceph-mon[29506]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:33.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:33 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:33.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:33 smithi111 ceph-mon[25603]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:36 smithi017 ceph-mon[24769]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:36 smithi017 ceph-mon[29506]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:36 smithi111 ceph-mon[25603]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:39:37] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-02-14T17:39:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:38 smithi017 ceph-mon[24769]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:38.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:38 smithi017 ceph-mon[29506]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:38 smithi111 ceph-mon[25603]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:39.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:40 smithi017 ceph-mon[24769]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:40 smithi017 ceph-mon[29506]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:40 smithi111 ceph-mon[25603]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:42 smithi017 ceph-mon[24769]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:42 smithi017 ceph-mon[29506]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:42 smithi111 ceph-mon[25603]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:44 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:44 smithi017 ceph-mon[24769]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:44 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:44 smithi017 ceph-mon[29506]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:44 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:44 smithi111 ceph-mon[25603]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:46.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:46 smithi111 ceph-mon[25603]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:46.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:46 smithi017 ceph-mon[24769]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:46.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:46 smithi017 ceph-mon[29506]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:39:47] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-02-14T17:39:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:48 smithi017 ceph-mon[24769]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:48 smithi017 ceph-mon[29506]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:48 smithi111 ceph-mon[25603]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:50 smithi017 ceph-mon[24769]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:50 smithi017 ceph-mon[29506]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:50 smithi111 ceph-mon[25603]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:52 smithi017 ceph-mon[24769]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:52 smithi017 ceph-mon[29506]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:52 smithi111 ceph-mon[25603]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:54 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:54 smithi017 ceph-mon[24769]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:54 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:54.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:54 smithi017 ceph-mon[29506]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:54.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:54 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:39:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:54 smithi111 ceph-mon[25603]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:39:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:56 smithi017 ceph-mon[24769]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:56 smithi017 ceph-mon[29506]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:56 smithi111 ceph-mon[25603]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:39:57.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:39:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:39:57] "GET /metrics HTTP/1.1" 200 37913 "" "Prometheus/2.33.4" 2024-02-14T17:39:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:39:58 smithi017 ceph-mon[24769]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:39:58 smithi017 ceph-mon[29506]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:39:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:39:58 smithi111 ceph-mon[25603]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:00 smithi017 ceph-mon[24769]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:00 smithi017 ceph-mon[24769]: overall HEALTH_OK 2024-02-14T17:40:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:00 smithi017 ceph-mon[29506]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:00 smithi017 ceph-mon[29506]: overall HEALTH_OK 2024-02-14T17:40:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:00 smithi111 ceph-mon[25603]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:00 smithi111 ceph-mon[25603]: overall HEALTH_OK 2024-02-14T17:40:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:02 smithi017 ceph-mon[24769]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:02 smithi017 ceph-mon[29506]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:02 smithi111 ceph-mon[25603]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:04 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:04 smithi017 ceph-mon[24769]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:04 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:04 smithi017 ceph-mon[29506]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:04 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:04 smithi111 ceph-mon[25603]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:06.680 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:06.681 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:06.681 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:06.681 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:06.681 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:06.681 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:06.681 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:06.681 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:06 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:06 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:40:07.607 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:40:07.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:07 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:40:07.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:07 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:40:07.677 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:07 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:40:07.678 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:07 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:40:07] "GET /metrics HTTP/1.1" 200 37913 "" "Prometheus/2.33.4" 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:40:07.832+0000 7fe2d57c2700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:07.988 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:07.989 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return future.result(timeout) 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.__get_result() 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:07.990 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: raise self._exception 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:07.991 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:07.992 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:08 smithi017 ceph-mon[24769]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:08 smithi017 ceph-mon[29506]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:08 smithi111 ceph-mon[25603]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:09 smithi017 ceph-mon[29506]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:09.691 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-02-14T17:40:09.692 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.693 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:09 smithi017 ceph-mon[24769]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:09.694 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:09.695 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:09.696 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:09.697 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:09.698 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:09 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:09 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:09.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:09 smithi111 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:09.734 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:09.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: return future.result(timeout) 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:09.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.__get_result() 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: raise self._exception 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:40:09.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:09.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:09 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:10 smithi017 ceph-mon[24769]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:10 smithi017 ceph-mon[29506]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:10 smithi111 ceph-mon[25603]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:12 smithi017 ceph-mon[24769]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:12 smithi017 ceph-mon[29506]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:12 smithi111 ceph-mon[25603]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:14 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:14 smithi017 ceph-mon[24769]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:14 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:14 smithi017 ceph-mon[29506]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:14 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:14 smithi111 ceph-mon[25603]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:40:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:16 smithi017 ceph-mon[24769]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:16 smithi017 ceph-mon[29506]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:16 smithi111 ceph-mon[25603]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:17.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:17 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:40:17] "GET /metrics HTTP/1.1" 200 37909 "" "Prometheus/2.33.4" 2024-02-14T17:40:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:18 smithi017 ceph-mon[24769]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:18 smithi017 ceph-mon[29506]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:18 smithi111 ceph-mon[25603]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:20 smithi017 ceph-mon[24769]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:20 smithi017 ceph-mon[29506]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:20 smithi111 ceph-mon[25603]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:22.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:22 smithi017 ceph-mon[24769]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:22 smithi017 ceph-mon[29506]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:22 smithi111 ceph-mon[25603]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:24 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:24 smithi017 ceph-mon[29506]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:24 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:24 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:24 smithi017 ceph-mon[24769]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:24.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:24 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:24 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:24 smithi111 ceph-mon[25603]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:24 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:26 smithi017 ceph-mon[24769]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:26 smithi017 ceph-mon[29506]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:26 smithi111 ceph-mon[25603]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:27.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:40:27] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.33.4" 2024-02-14T17:40:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:28 smithi017 ceph-mon[24769]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:28 smithi017 ceph-mon[29506]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:28 smithi111 ceph-mon[25603]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:30.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:30 smithi017 ceph-mon[24769]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:30 smithi017 ceph-mon[29506]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:30 smithi111 ceph-mon[25603]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:31 smithi017 ceph-mon[24769]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:31.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:31 smithi017 ceph-mon[29506]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:31 smithi111 ceph-mon[25603]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:33.842 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:33 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:33.842 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:33 smithi017 ceph-mon[24769]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:33 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:33.843 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:33 smithi017 ceph-mon[29506]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:33.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:33 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:33.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:33 smithi111 ceph-mon[25603]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:36 smithi017 ceph-mon[24769]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:36 smithi017 ceph-mon[29506]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:36 smithi111 ceph-mon[25603]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:37.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:40:37] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.33.4" 2024-02-14T17:40:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:38 smithi017 ceph-mon[24769]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:38 smithi017 ceph-mon[29506]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:38 smithi111 ceph-mon[25603]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:39.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:39 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:39 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:39 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:40 smithi017 ceph-mon[24769]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:40 smithi017 ceph-mon[29506]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:40 smithi111 ceph-mon[25603]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:41.942 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:40:42.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:42 smithi017 ceph-mon[24769]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:42.558 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:42 smithi017 ceph-mon[29506]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:42 smithi111 ceph-mon[25603]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:44.105 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (12m) 2m ago 20m 24.4M - 0.23.0 ba2b418f427c 556f84c8c0ad 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (11m) 2m ago 20m 56.2M - 8.3.5 dad864ee21e9 3af96da5f3f2 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (5m) 2m ago 19m 49.0M - 3.5 e1d6a67b021e 8ecd81b68f43 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283 running (15m) 2m ago 24m 493M - 19.0.0-1430-g8727ce93 170846cf6432 2ffaf8e8fceb 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (9m) 2m ago 26m 75.6M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (26m) 2m ago 26m 66.8M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (24m) 2m ago 24m 54.4M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (25m) 2m ago 25m 47.5M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (12m) 2m ago 20m 17.3M - 1.3.1 1dbe0e931976 f9e29bdc31de 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (12m) 2m ago 20m 21.8M - 1.3.1 1dbe0e931976 5d05f9239e5a 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (24m) 2m ago 24m 53.8M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (24m) 2m ago 24m 53.1M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (23m) 2m ago 23m 51.8M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:40:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (23m) 2m ago 23m 50.5M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:40:44.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (22m) 2m ago 22m 52.6M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:40:44.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (22m) 2m ago 22m 54.2M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:40:44.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (22m) 2m ago 22m 49.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:40:44.107 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (21m) 2m ago 21m 55.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:40:44.107 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (2m) 2m ago 21m 44.6M - 2.33.4 514e6a882f6e 8006aec214ef 2024-02-14T17:40:44.107 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (19m) 2m ago 19m 76.4M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:40:44.107 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (19m) 2m ago 19m 76.7M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:40:44.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:44 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:44.409 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:44 smithi017 ceph-mon[24769]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:44.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:44 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:44.409 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:44 smithi017 ceph-mon[29506]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:44.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ls' 2024-02-14T17:40:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:44 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:44 smithi111 ceph-mon[25603]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:45.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:45 smithi017 ceph-mon[24769]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:45.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:45 smithi017 ceph-mon[29506]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:45.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:45 smithi111 ceph-mon[25603]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:46.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:46 smithi017 ceph-mon[24769]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:46.374 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:46 smithi017 ceph-mon[29506]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:46.635 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:46 smithi111 ceph-mon[25603]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager ?:9093,9094 1/1 3m ago 21m smithi017=a;count:1 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:grafana ?:3000 1/1 2m ago 21m smithi111=a;count:1 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo ?:5000 1/1 3m ago 19m count:1 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:mgr 2/2 3m ago 24m smithi017=y;smithi111=x;count:2 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:mon 3/3 3m ago 25m smithi017:172.21.15.17=a;smithi017:[v2:172.21.15.17:3301,v1:172.21.15.17:6790]=c;smithi111:172.21.15.111=b;count:3 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter ?:9100 2/2 3m ago 21m smithi017=a;smithi111=b;count:2 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:osd 8 3m ago - 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus ?:9095 1/1 2m ago 21m smithi111=a;count:1 2024-02-14T17:40:46.827 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo ?:8000 2/2 3m ago 19m count:2 2024-02-14T17:40:47.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions' 2024-02-14T17:40:47.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:40:47] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-02-14T17:40:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:48 smithi017 ceph-mon[24769]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:48 smithi017 ceph-mon[24769]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:48 smithi017 ceph-mon[29506]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:48 smithi017 ceph-mon[29506]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:48 smithi111 ceph-mon[25603]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:48 smithi111 ceph-mon[25603]: from='client.15396 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:49.490 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:40:49.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:49.492 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:40:49.492 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:40:49.492 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:40:49.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:40:49.492 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:40:50.094 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-14T17:40:50.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:50 smithi017 ceph-mon[24769]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:50.639 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:50 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2595546044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:40:50.639 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:50 smithi017 ceph-mon[29506]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:50.639 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:50 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2595546044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:40:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:50 smithi111 ceph-mon[25603]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:50 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2595546044' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:40:52.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:52 smithi017 ceph-mon[24769]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:52.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:52 smithi017 ceph-mon[29506]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:52.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:52 smithi111 ceph-mon[25603]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:53 smithi017 ceph-mon[24769]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:53.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:53 smithi017 ceph-mon[29506]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:53 smithi111 ceph-mon[25603]: from='client.15408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:54.107 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:40:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:54.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:54.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:54 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:54.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:40:54.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:54 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:40:54.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-14T17:40:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:55 smithi017 ceph-mon[24769]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:40:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:55 smithi017 ceph-mon[24769]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:40:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:55 smithi017 ceph-mon[24769]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:40:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:55 smithi017 ceph-mon[29506]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:40:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:55 smithi017 ceph-mon[29506]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:40:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:55 smithi017 ceph-mon[29506]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:40:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:55 smithi111 ceph-mon[25603]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:40:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:55 smithi111 ceph-mon[25603]: Reconfiguring iscsi.foo.smithi017.wtnmjz (dependencies changed)... 2024-02-14T17:40:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:55 smithi111 ceph-mon[25603]: Reconfiguring daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:40:56.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:56 smithi017 ceph-mon[24769]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:56.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:56 smithi017 ceph-mon[29506]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:56 smithi111 ceph-mon[25603]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:40:57.335+0000 7fe2d57c2700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:57.608 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:57.609 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return future.result(timeout) 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return self.__get_result() 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: raise self._exception 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:57.610 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:57.611 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Creating ceph-iscsi config... 2024-02-14T17:40:57.611 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:57.611 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:57.611 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:57.611 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:57.611 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:57.611 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: Traceback (most recent call last): 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: return _run_code(code, main_globals, None, 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: exec(code, run_globals) 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:57.612 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:57.613 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:40:57 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:40:57] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-02-14T17:40:57.704 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:40:58.388 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:40:58.388 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (12m) 3m ago 20m 24.4M - 0.23.0 ba2b418f427c 556f84c8c0ad 2024-02-14T17:40:58.388 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (12m) 2m ago 20m 56.2M - 8.3.5 dad864ee21e9 3af96da5f3f2 2024-02-14T17:40:58.388 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 running (6m) 3m ago 19m 49.0M - 3.5 e1d6a67b021e 8ecd81b68f43 2024-02-14T17:40:58.388 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283 running (15m) 2m ago 24m 493M - 19.0.0-1430-g8727ce93 170846cf6432 2ffaf8e8fceb 2024-02-14T17:40:58.388 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (9m) 3m ago 26m 75.6M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (26m) 3m ago 26m 66.8M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (25m) 2m ago 25m 54.4M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (25m) 3m ago 25m 47.5M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (12m) 3m ago 20m 17.3M - 1.3.1 1dbe0e931976 f9e29bdc31de 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (12m) 2m ago 20m 21.8M - 1.3.1 1dbe0e931976 5d05f9239e5a 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (24m) 3m ago 24m 53.8M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (24m) 3m ago 24m 53.1M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (23m) 3m ago 23m 51.8M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (23m) 3m ago 23m 50.5M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (23m) 2m ago 23m 52.6M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (22m) 2m ago 22m 54.2M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (22m) 2m ago 22m 49.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (22m) 2m ago 22m 55.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:40:58.389 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (3m) 2m ago 21m 44.6M - 2.33.4 514e6a882f6e 8006aec214ef 2024-02-14T17:40:58.390 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (19m) 3m ago 19m 76.4M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:40:58.390 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (19m) 2m ago 19m 76.7M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:40:58.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:58 smithi017 ceph-mon[24769]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:58.601 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:58 smithi017 ceph-mon[24769]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:58.601 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:58.602 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: return future.result(timeout) 2024-02-14T17:40:58.603 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: return self.__get_result() 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: raise self._exception 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:58.604 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.605 INFO:journalctl@ceph.mon.a.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:58 smithi017 ceph-mon[24769]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:58 smithi017 ceph-mon[29506]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:58 smithi017 ceph-mon[29506]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:58.606 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.607 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.607 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.607 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:58.607 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:58.607 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:58.607 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:58.608 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:58.608 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:58.608 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:58.608 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:58.608 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:58.608 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:58.609 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:58.609 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.609 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.609 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.609 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:58.609 INFO:journalctl@ceph.mon.c.smithi017.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:58.609 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:58.610 INFO:journalctl@ceph.mon.c.smithi017.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:58.610 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:58.610 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:58.610 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:58.610 INFO:journalctl@ceph.mon.c.smithi017.stdout: return future.result(timeout) 2024-02-14T17:40:58.610 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:58.611 INFO:journalctl@ceph.mon.c.smithi017.stdout: return self.__get_result() 2024-02-14T17:40:58.611 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:58.611 INFO:journalctl@ceph.mon.c.smithi017.stdout: raise self._exception 2024-02-14T17:40:58.611 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:58.611 INFO:journalctl@ceph.mon.c.smithi017.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:58.611 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:58.611 INFO:journalctl@ceph.mon.c.smithi017.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:58.612 INFO:journalctl@ceph.mon.c.smithi017.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:58.612 INFO:journalctl@ceph.mon.c.smithi017.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:58.612 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:58.612 INFO:journalctl@ceph.mon.c.smithi017.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:58.612 INFO:journalctl@ceph.mon.c.smithi017.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:58.612 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.613 INFO:journalctl@ceph.mon.c.smithi017.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.613 INFO:journalctl@ceph.mon.c.smithi017.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.613 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:58.613 INFO:journalctl@ceph.mon.c.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:58.613 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:58.613 INFO:journalctl@ceph.mon.c.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:40:58.613 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:58.614 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:58.614 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:58.614 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:58.614 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:58.614 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:58.614 INFO:journalctl@ceph.mon.c.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:58.615 INFO:journalctl@ceph.mon.c.smithi017.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.615 INFO:journalctl@ceph.mon.c.smithi017.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.615 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:58 smithi017 ceph-mon[29506]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:40:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:58 smithi111 ceph-mon[25603]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:58 smithi111 ceph-mon[25603]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:58.735 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2372, in _daemon_action 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-02-14T17:40:58.736 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 739, in wait_async 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.event_loop.get_result(coro, timeout) 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: return future.result(timeout) 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: return self.__get_result() 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: raise self._exception 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: config_blobs=daemon_spec.final_config, 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Creating ceph-iscsi config... 2024-02-14T17:40:58.737 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Non-zero exit code 1 from systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: Traceback (most recent call last): 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: exec(code, run_globals) 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:40:58.738 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1065, in deploy_daemon 2024-02-14T17:40:58.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:40:58.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: RuntimeError: Failed command: systemctl restart ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:40:58.739 INFO:journalctl@ceph.mon.b.smithi111.stdout: See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:40:58.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:58 smithi111 ceph-mon[25603]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:40:58.791 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:40:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:40:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:40:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:40:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:40:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:40:58.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:40:58.792 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:40:59.136 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:40:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", 2024-02-14T17:40:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:40:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-14T17:40:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:40:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "", 2024-02-14T17:40:59.136 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 image", 2024-02-14T17:40:59.137 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:40:59.137 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:40:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:59 smithi017 ceph-mon[24769]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:59 smithi017 ceph-mon[24769]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:40:59 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3866418673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:40:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:59 smithi017 ceph-mon[29506]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:59 smithi017 ceph-mon[29506]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:40:59 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3866418673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:40:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:59 smithi111 ceph-mon[25603]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:59 smithi111 ceph-mon[25603]: from='client.25336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:40:59.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:40:59 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3866418673' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:00.621 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:00.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:00.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:00.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:41:00.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:00.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-14T17:41:00.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-14T17:41:00.622 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[24769]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:41:00] ENGINE Bus STOPPING 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:41:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:41:00] ENGINE Bus STOPPED 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:41:00.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:00.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-14T17:41:00.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-14T17:41:00.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-mon[29506]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: from='mgr.24956 172.21.15.111:0/4210925296' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-02-14T17:41:00.683 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mon[25603]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:41:00.684 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:00.520+0000 7fe3430cb700 -1 mgr handle_mgr_map I was active but no longer am 2024-02-14T17:41:00.684 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ignoring --setuser ceph since I am not root 2024-02-14T17:41:00.684 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: ignoring --setgroup ceph since I am not root 2024-02-14T17:41:00.684 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:00 smithi111 ceph-mgr[45832]: -- 172.21.15.111:0/275436204 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c7b31261a0 con 0x55c7b30fd000 2024-02-14T17:41:00.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:41:00] ENGINE Bus STARTING 2024-02-14T17:41:00.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:41:00] ENGINE Serving on http://:::9283 2024-02-14T17:41:00.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:41:00] ENGINE Bus STARTED 2024-02-14T17:41:01.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:01.103+0000 7fe216511200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:41:01.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:01.200+0000 7fe216511200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: Failing over to other MGR 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.24956 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: mgrmap e36: y(active, starting, since 0.994729s) 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:41:01.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: Standby manager daemon x started 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: Manager daemon y is now available 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:01.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:41:01.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:41:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: Failing over to other MGR 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.24956 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: mgrmap e36: y(active, starting, since 0.994729s) 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:41:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:41:01.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:41:01.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: Standby manager daemon x started 2024-02-14T17:41:01.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:41:01.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:41:01.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:41:01.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:41:01.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:41:01.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: Manager daemon y is now available 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:41:01.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: Upgrade: Need to upgrade myself (mgr.x) 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: Failing over to other MGR 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.24956 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: mgrmap e36: y(active, starting, since 0.994729s) 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:41:01.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: Manager daemon y is now available 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:41:01.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:41:02.126 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:01.886+0000 7fe216511200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:41:02.126 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:01 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:01.981+0000 7fe216511200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:41:02.126 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:02.124+0000 7fe216511200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:41:02.460 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:02.243+0000 7fe216511200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:41:02.460 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:02.339+0000 7fe216511200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:41:02.716 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-mon[25603]: mgrmap e37: y(active, since 2s), standbys: x 2024-02-14T17:41:02.716 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:41:02.716 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-mon[25603]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:02.717 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:02.618+0000 7fe216511200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:41:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:02 smithi017 ceph-mon[24769]: mgrmap e37: y(active, since 2s), standbys: x 2024-02-14T17:41:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:02 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:41:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:02 smithi017 ceph-mon[24769]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:02.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:02 smithi017 ceph-mon[29506]: mgrmap e37: y(active, since 2s), standbys: x 2024-02-14T17:41:02.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:02 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:41:02.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:02 smithi017 ceph-mon[29506]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:02.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:02.728+0000 7fe216511200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:02.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:02.816+0000 7fe216511200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:41:02.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:02.914+0000 7fe216511200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:41:03.376 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:03 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:03.121+0000 7fe216511200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:41:03.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[24769]: [14/Feb/2024:17:41:01] ENGINE Bus STARTING 2024-02-14T17:41:03.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[24769]: [14/Feb/2024:17:41:01] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:41:03.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[24769]: [14/Feb/2024:17:41:01] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[24769]: [14/Feb/2024:17:41:01] ENGINE Bus STARTED 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[24769]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[29506]: [14/Feb/2024:17:41:01] ENGINE Bus STARTING 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[29506]: [14/Feb/2024:17:41:01] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[29506]: [14/Feb/2024:17:41:01] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[29506]: [14/Feb/2024:17:41:01] ENGINE Bus STARTED 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:03.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:03 smithi017 ceph-mon[29506]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:03 smithi111 ceph-mon[25603]: [14/Feb/2024:17:41:01] ENGINE Bus STARTING 2024-02-14T17:41:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:03 smithi111 ceph-mon[25603]: [14/Feb/2024:17:41:01] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:41:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:03 smithi111 ceph-mon[25603]: [14/Feb/2024:17:41:01] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:41:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:03 smithi111 ceph-mon[25603]: [14/Feb/2024:17:41:01] ENGINE Bus STARTED 2024-02-14T17:41:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:03 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:03.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:03 smithi111 ceph-mon[25603]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:04.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:04.144+0000 7fe216511200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:41:04.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:04.277+0000 7fe216511200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:41:04.927 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:04 smithi017 ceph-mon[24769]: mgrmap e38: y(active, since 4s), standbys: x 2024-02-14T17:41:04.927 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:04 smithi017 ceph-mon[29506]: mgrmap e38: y(active, since 4s), standbys: x 2024-02-14T17:41:04.928 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:04 smithi111 ceph-mon[25603]: mgrmap e38: y(active, since 4s), standbys: x 2024-02-14T17:41:04.929 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:04.623+0000 7fe216511200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:41:04.929 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:04.715+0000 7fe216511200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:41:05.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:04 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:04.927+0000 7fe216511200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:05.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:05.120+0000 7fe216511200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:41:05.713 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:05.426+0000 7fe216511200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:05.713 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-mon[25603]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:05.713 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.713 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.713 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.714 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[24769]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:05.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[29506]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:05.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:05 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:05.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:05.711+0000 7fe216511200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:06.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:05 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:05.992+0000 7fe216511200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:41:07.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:06 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:06.859+0000 7fe216511200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:41:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:07.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:41:07.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:41:07.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:07.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:07 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:07.463+0000 7fe216511200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:41:07.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:07.566+0000 7fe216511200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:41:07.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:07.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:07.786+0000 7fe216511200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:41:08.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:41:07] ENGINE Bus STARTING 2024-02-14T17:41:08.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: CherryPy Checker: 2024-02-14T17:41:08.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: The Application mounted at '' has an empty config. 2024-02-14T17:41:08.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: 2024-02-14T17:41:08.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:41:07] ENGINE Serving on http://:::9283 2024-02-14T17:41:08.235 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:07 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[45828]: [14/Feb/2024:17:41:07] ENGINE Bus STARTED 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 142 op/s 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-14T17:41:08.558 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:41:08.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Standby manager daemon x restarted 2024-02-14T17:41:08.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: Standby manager daemon x started 2024-02-14T17:41:08.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:08.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:08.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:08.559 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:08 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 142 op/s 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Standby manager daemon x restarted 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: Standby manager daemon x started 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:08.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 0 B/s wr, 142 op/s 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-14T17:41:08.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Standby manager daemon x restarted 2024-02-14T17:41:08.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:41:08.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:08.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:08.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:08.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:08 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/2387021873' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:09.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:09 smithi111 ceph-mon[25603]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:41:09.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:09 smithi111 ceph-mon[25603]: mgrmap e39: y(active, since 8s), standbys: x 2024-02-14T17:41:09.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:09 smithi017 ceph-mon[24769]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:41:09.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:09 smithi017 ceph-mon[24769]: mgrmap e39: y(active, since 8s), standbys: x 2024-02-14T17:41:09.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:09 smithi017 ceph-mon[29506]: Reconfiguring daemon prometheus.a on smithi111 2024-02-14T17:41:09.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:09 smithi017 ceph-mon[29506]: mgrmap e39: y(active, since 8s), standbys: x 2024-02-14T17:41:10.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:10 smithi111 ceph-mon[25603]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 106 op/s 2024-02-14T17:41:10.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:10 smithi017 ceph-mon[24769]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 106 op/s 2024-02-14T17:41:10.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:10 smithi017 ceph-mon[29506]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 106 op/s 2024-02-14T17:41:11.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:10 smithi111 systemd[1]: Stopping Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:11.118 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:10 smithi111 bash[60657]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:41:11.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:41:11.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:41:11.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:41:11.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-14T17:41:11.377 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.117Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.118Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.118Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.118Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[57857]: ts=2024-02-14T17:41:11.118Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-14T17:41:11.378 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 podman[60670]: 2024-02-14 17:41:11.127415269 +0000 UTC m=+0.132509350 container died 8006aec214efb08dabfb8565436bafa38a4aea403624d439504f85d1f60bc973 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:11.734 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 podman[60670]: 2024-02-14 17:41:11.47545145 +0000 UTC m=+0.480545540 container cleanup 8006aec214efb08dabfb8565436bafa38a4aea403624d439504f85d1f60bc973 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:11.734 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 bash[60670]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:41:11.734 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 bash[60698]: Error: no container with name or ID "ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus.a" found: no such container 2024-02-14T17:41:12.112 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 podman[60684]: 2024-02-14 17:41:11.777297766 +0000 UTC m=+0.651607972 container remove 8006aec214efb08dabfb8565436bafa38a4aea403624d439504f85d1f60bc973 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:12.113 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-14T17:41:12.113 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 systemd[1]: Stopped Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:12.113 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Consumed 1.293s CPU time. 2024-02-14T17:41:12.113 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:11 smithi111 systemd[1]: Starting Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:12.113 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 podman[60766]: 2024-02-14 17:41:11.906946285 +0000 UTC m=+0.019045098 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-02-14T17:41:12.113 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 podman[60766]: 2024-02-14 17:41:12.111687738 +0000 UTC m=+0.223786542 container create 6143817423aafe5b51aebff66a81b4557617e055dd885c6bf122a316280db5ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:12.365 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 podman[60766]: 2024-02-14 17:41:12.338870496 +0000 UTC m=+0.450969310 container init 6143817423aafe5b51aebff66a81b4557617e055dd885c6bf122a316280db5ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:12.365 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 podman[60766]: 2024-02-14 17:41:12.341590535 +0000 UTC m=+0.453689341 container start 6143817423aafe5b51aebff66a81b4557617e055dd885c6bf122a316280db5ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-mon[25603]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 84 op/s 2024-02-14T17:41:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.363Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.364Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.364Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.364Z caller=main.go:518 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi111 (none))" 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.364Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.364Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.365Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.366Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.366Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.368Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.369Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=619.92µs 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.369Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:41:12.735 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 bash[60766]: 6143817423aafe5b51aebff66a81b4557617e055dd885c6bf122a316280db5ce 2024-02-14T17:41:12.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 systemd[1]: Started Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:12.736 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.576Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-02-14T17:41:12.858 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:12 smithi017 ceph-mon[24769]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 84 op/s 2024-02-14T17:41:12.858 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:12 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:12.858 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:12 smithi017 ceph-mon[29506]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 84 op/s 2024-02-14T17:41:12.858 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:12 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:13.096 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.798Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-02-14T17:41:13.096 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.842Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-02-14T17:41:13.096 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.881Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.884Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.884Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.884Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=55.822µs wal_replay_duration=515.618031ms total_replay_duration=516.314411ms 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.888Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.888Z caller=main.go:947 level=info msg="TSDB started" 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.888Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.909Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.791605ms db_storage=828ns remote_storage=1.57µs web_handler=451ns query_engine=685ns scrape=393.919µs scrape_sd=88.352µs notify=15.41µs notify_sd=17.774µs rules=20.925284ms 2024-02-14T17:41:13.097 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:12 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:12.910Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-02-14T17:41:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:13 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:13 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:13 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:41:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:13 smithi111 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:41:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:13 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:13.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:13.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:13.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:41:13.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[24769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:41:13.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:13.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:13.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:13.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:41:13.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[29506]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:41:13.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:13 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:14.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:14 smithi111 ceph-mon[25603]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-14T17:41:14.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:14 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:41:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:14 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:41:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:14 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:14.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[24769]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-14T17:41:14.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:41:14.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:41:14.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:14.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[29506]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-14T17:41:14.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:41:14.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:41:14.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:14 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:15.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:15 smithi111 ceph-mon[25603]: Upgrade: Updating mgr.x 2024-02-14T17:41:15.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:15 smithi111 ceph-mon[25603]: Deploying daemon mgr.x on smithi111 2024-02-14T17:41:15.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:15 smithi017 ceph-mon[29506]: Upgrade: Updating mgr.x 2024-02-14T17:41:15.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:15 smithi017 ceph-mon[29506]: Deploying daemon mgr.x on smithi111 2024-02-14T17:41:15.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:15 smithi017 ceph-mon[24769]: Upgrade: Updating mgr.x 2024-02-14T17:41:15.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:15 smithi017 ceph-mon[24769]: Deploying daemon mgr.x on smithi111 2024-02-14T17:41:16.398 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:16 smithi111 systemd[1]: Stopping Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:16.651 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:16 smithi111 podman[61118]: 2024-02-14 17:41:16.397309581 +0000 UTC m=+0.119400296 container died 2ffaf8e8fcebcc1093c727b93a86bc2362f13aa4619222deaca6b3ae20792a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-14T17:41:16.652 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:16 smithi111 ceph-mon[25603]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-14T17:41:16.652 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:16 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:16.652 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:16 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:16.918 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:16 smithi111 podman[61118]: 2024-02-14 17:41:16.724931519 +0000 UTC m=+0.447022204 container cleanup 2ffaf8e8fcebcc1093c727b93a86bc2362f13aa4619222deaca6b3ae20792a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-14T17:41:16.918 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:16 smithi111 bash[61118]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x 2024-02-14T17:41:16.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:16 smithi017 ceph-mon[24769]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-14T17:41:16.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:16 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:16.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:16 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:16.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:16 smithi017 ceph-mon[29506]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-14T17:41:16.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:16 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:16.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:16 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:17.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:16 smithi111 podman[61132]: 2024-02-14 17:41:16.917696463 +0000 UTC m=+0.517875482 container remove 2ffaf8e8fcebcc1093c727b93a86bc2362f13aa4619222deaca6b3ae20792a4b (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-14T17:41:17.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:16 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:41:17.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-02-14T17:41:17.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 systemd[1]: Stopped Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:17.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service: Consumed 57.756s CPU time. 2024-02-14T17:41:17.570 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 systemd[1]: Starting Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:17.570 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 podman[61265]: 2024-02-14 17:41:17.433135836 +0000 UTC m=+0.019736099 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:41:17.864 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 podman[61265]: 2024-02-14 17:41:17.569279495 +0000 UTC m=+0.155879751 container create 8d2779e92e1b144c8bef9bba76f9cad4a3e34a2b4efa1a7439b01c96e1ef0a30 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-14T17:41:17.864 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 podman[61265]: 2024-02-14 17:41:17.788466878 +0000 UTC m=+0.375067132 container init 8d2779e92e1b144c8bef9bba76f9cad4a3e34a2b4efa1a7439b01c96e1ef0a30 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214) 2024-02-14T17:41:17.864 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 podman[61265]: 2024-02-14 17:41:17.791142444 +0000 UTC m=+0.377742699 container start 8d2779e92e1b144c8bef9bba76f9cad4a3e34a2b4efa1a7439b01c96e1ef0a30 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-14T17:41:17.864 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 ceph-mgr[61283]: -- 172.21.15.111:0/1233759948 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561aba9ec1a0 con 0x561aba9c3000 2024-02-14T17:41:17.865 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 bash[61265]: 8d2779e92e1b144c8bef9bba76f9cad4a3e34a2b4efa1a7439b01c96e1ef0a30 2024-02-14T17:41:18.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:17 smithi111 systemd[1]: Started Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:18 smithi111 ceph-mon[25603]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-02-14T17:41:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:18 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:18 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:18 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:18.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:18.357+0000 7f5703281200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:41:18.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:18.470+0000 7f5703281200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:41:18.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[29506]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-02-14T17:41:18.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:18.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:18.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:18.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[24769]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-02-14T17:41:18.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:18.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:18.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:18 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:19.404 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:19.147+0000 7f5703281200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:41:19.404 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:19.255+0000 7f5703281200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:41:19.712 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:19.402+0000 7f5703281200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:41:19.712 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:19.505+0000 7f5703281200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:41:19.712 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:19.593+0000 7f5703281200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:41:19.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:19.859+0000 7f5703281200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:41:20.389 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:19 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:19.996+0000 7f5703281200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:20.389 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:20.104+0000 7f5703281200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:41:20.389 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:20.221+0000 7f5703281200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:41:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:20 smithi111 ceph-mon[25603]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:20.735 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:20 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:20.400+0000 7f5703281200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:41:20.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:20 smithi017 ceph-mon[24769]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:20 smithi017 ceph-mon[29506]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:21.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:21.402+0000 7f5703281200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:41:21.880 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:21.535+0000 7f5703281200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:41:22.183 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:21.879+0000 7f5703281200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:41:22.183 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:21.981+0000 7f5703281200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:41:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:41:21] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.33.4" 2024-02-14T17:41:22.471 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:22.181+0000 7f5703281200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:22.471 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:22.388+0000 7f5703281200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:41:22.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:22.696+0000 7f5703281200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:41:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:22 smithi111 ceph-mon[25603]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:22 smithi017 ceph-mon[24769]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:22.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:22.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:22 smithi017 ceph-mon[29506]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:22.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:22.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:23.228 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:22.972+0000 7f5703281200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:41:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:23 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:23.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:23.226+0000 7f5703281200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:41:23.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:23 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:23.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:23 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:24.420 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:24.102+0000 7f5703281200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:41:24.687 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:24.685+0000 7f5703281200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:41:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:24 smithi111 ceph-mon[25603]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:24 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:24.687 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:24 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:24 smithi017 ceph-mon[24769]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:24 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:24.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:24 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:24.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:24 smithi017 ceph-mon[29506]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:41:24.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:24 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:24.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:24 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:24.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:24.788+0000 7f5703281200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:41:25.306 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:25.013+0000 7f5703281200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:41:25.306 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: [14/Feb/2024:17:41:25] ENGINE Bus STARTING 2024-02-14T17:41:25.306 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: CherryPy Checker: 2024-02-14T17:41:25.306 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: The Application mounted at '' has an empty config. 2024-02-14T17:41:25.306 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:41:25.306 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: [14/Feb/2024:17:41:25] ENGINE Serving on http://:::9283 2024-02-14T17:41:25.307 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: [14/Feb/2024:17:41:25] ENGINE Bus STARTED 2024-02-14T17:41:25.627 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-mon[25603]: Standby manager daemon x restarted 2024-02-14T17:41:25.627 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-mon[25603]: Standby manager daemon x started 2024-02-14T17:41:25.627 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:25.627 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:25.627 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:25.628 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:25 smithi111 ceph-mon[25603]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:25.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[24769]: Standby manager daemon x restarted 2024-02-14T17:41:25.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[24769]: Standby manager daemon x started 2024-02-14T17:41:25.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:25.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:25.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:25.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[24769]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:25.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[29506]: Standby manager daemon x restarted 2024-02-14T17:41:25.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:41:25.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:41:25.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:41:25.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:41:25.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:25 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/3668350678' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:41:26.848 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:26 smithi111 ceph-mon[25603]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-02-14T17:41:26.848 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:26 smithi111 ceph-mon[25603]: mgrmap e40: y(active, since 26s), standbys: x 2024-02-14T17:41:26.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:26 smithi017 ceph-mon[24769]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-02-14T17:41:26.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:26 smithi017 ceph-mon[24769]: mgrmap e40: y(active, since 26s), standbys: x 2024-02-14T17:41:26.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:26 smithi017 ceph-mon[29506]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 48 op/s 2024-02-14T17:41:26.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:26 smithi017 ceph-mon[29506]: mgrmap e40: y(active, since 26s), standbys: x 2024-02-14T17:41:28.006 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all mgr 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-14T17:41:28.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all crash 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all mds 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.008 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all nfs 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:28.009 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all mgr 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all crash 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all mds 2024-02-14T17:41:28.010 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.011 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all nfs 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:41:28.012 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:27 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 51 op/s 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:28.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all mgr 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all crash 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all mds 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:41:28.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all nfs 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:41:28.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:27 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:29.507 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:29 smithi017 systemd[1]: Stopping Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:29.507 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:41:29.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:29 smithi017 podman[76626]: 2024-02-14 17:41:29.648665993 +0000 UTC m=+0.122967799 container died f9e29bdc31dee85ef65f67417f71eadc13b79b51eaab900c8db4c7feda948bc3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:30.184 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:41:30.184 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (13m) 24s ago 21m 24.1M - 0.23.0 ba2b418f427c 556f84c8c0ad 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (12m) 7s ago 20m 57.3M - 8.3.5 dad864ee21e9 3af96da5f3f2 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 unknown 24s ago 20m 54.9M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (12s) 7s ago 25m 66.5M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (10m) 24s ago 27m 474M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (27m) 24s ago 27m 75.1M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (25m) 7s ago 25m 60.2M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (25m) 24s ago 25m 53.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (13m) 24s ago 21m 17.9M - 1.3.1 1dbe0e931976 f9e29bdc31de 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (12m) 7s ago 21m 19.2M - 1.3.1 1dbe0e931976 5d05f9239e5a 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (25m) 24s ago 25m 53.1M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (24m) 24s ago 24m 53.7M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (24m) 24s ago 24m 51.1M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (24m) 24s ago 24m 50.7M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:41:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (23m) 7s ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:41:30.186 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (23m) 7s ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:41:30.186 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (22m) 7s ago 22m 51.4M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:41:30.186 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (22m) 7s ago 22m 56.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:41:30.186 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (18s) 7s ago 21m 52.3M - 2.33.4 514e6a882f6e 6143817423aa 2024-02-14T17:41:30.186 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (20m) 24s ago 20m 77.1M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:41:30.186 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (20m) 7s ago 20m 77.8M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:41:30.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:29 smithi111 ceph-mon[25603]: Upgrade: Updating node-exporter.a (1/2) 2024-02-14T17:41:30.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:29 smithi111 ceph-mon[25603]: Deploying daemon node-exporter.a on smithi017 2024-02-14T17:41:30.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:29 smithi111 ceph-mon[25603]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:30.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:29 smithi111 ceph-mon[25603]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:30.397 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[24769]: Upgrade: Updating node-exporter.a (1/2) 2024-02-14T17:41:30.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[24769]: Deploying daemon node-exporter.a on smithi017 2024-02-14T17:41:30.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[24769]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:30.398 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[24769]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:30.398 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[29506]: Upgrade: Updating node-exporter.a (1/2) 2024-02-14T17:41:30.398 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[29506]: Deploying daemon node-exporter.a on smithi017 2024-02-14T17:41:30.398 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[29506]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:30.398 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:29 smithi017 ceph-mon[29506]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:30.398 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 podman[76626]: 2024-02-14 17:41:30.162250765 +0000 UTC m=+0.636552559 container cleanup f9e29bdc31dee85ef65f67417f71eadc13b79b51eaab900c8db4c7feda948bc3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:30.399 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 bash[76626]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a 2024-02-14T17:41:30.611 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:41:30.612 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:41:30.613 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:41:30.648 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 podman[76658]: 2024-02-14 17:41:30.396896319 +0000 UTC m=+0.749965992 container remove f9e29bdc31dee85ef65f67417f71eadc13b79b51eaab900c8db4c7feda948bc3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:30.648 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:41:30.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-14T17:41:30.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 systemd[1]: Stopped Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:30.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Consumed 5.188s CPU time. 2024-02-14T17:41:30.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:30 smithi017 systemd[1]: Starting Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/2 daemons upgraded", 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading node-exporter daemons", 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:41:31.006 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:41:31.303 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:31 smithi017 bash[76858]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-14T17:41:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[24769]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[24769]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3742343943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:31.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[29506]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:31.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[29506]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:31.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3742343943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:31.691 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:31 smithi017 bash[76858]: Getting image source signatures 2024-02-14T17:41:31.691 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:31 smithi017 bash[76858]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-14T17:41:31.691 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:31 smithi017 bash[76858]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-14T17:41:31.691 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:31 smithi017 bash[76858]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-14T17:41:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:31 smithi111 ceph-mon[25603]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:31 smithi111 ceph-mon[25603]: from='client.15489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:31 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3742343943' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:31 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:32.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:41:31] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-02-14T17:41:32.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:32 smithi017 ceph-mon[24769]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:32.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:32 smithi017 ceph-mon[24769]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:32.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:32 smithi017 ceph-mon[29506]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:32.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:32 smithi017 ceph-mon[29506]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:32 smithi111 ceph-mon[25603]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:32 smithi111 ceph-mon[25603]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:41:33.309 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 bash[76858]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-14T17:41:33.309 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 bash[76858]: Writing manifest to image destination 2024-02-14T17:41:33.663 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:33.663 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:33.663 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 podman[76858]: 2024-02-14 17:41:33.475522319 +0000 UTC m=+2.482905044 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-14T17:41:33.663 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 podman[76858]: 2024-02-14 17:41:33.556826623 +0000 UTC m=+2.564209342 container create 3ed6d4a4c5ad2dbc911bc3d2fbb9a2fa17f2766a451cda0324c97b5d6a77e68f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:33 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:33.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 podman[76858]: 2024-02-14 17:41:33.775661303 +0000 UTC m=+2.783044024 container init 3ed6d4a4c5ad2dbc911bc3d2fbb9a2fa17f2766a451cda0324c97b5d6a77e68f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:33.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 podman[76858]: 2024-02-14 17:41:33.778425115 +0000 UTC m=+2.785807833 container start 3ed6d4a4c5ad2dbc911bc3d2fbb9a2fa17f2766a451cda0324c97b5d6a77e68f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:33.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.782Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-14T17:41:33.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.782Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-14T17:41:33.940 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=arp 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-14T17:41:33.941 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=edac 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-14T17:41:33.942 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=os 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-14T17:41:33.943 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=stat 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=time 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=uname 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-14T17:41:33.944 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-14T17:41:33.945 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.783Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-14T17:41:33.945 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.784Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-14T17:41:33.945 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a[76923]: ts=2024-02-14T17:41:33.784Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-14T17:41:33.945 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 bash[76858]: 3ed6d4a4c5ad2dbc911bc3d2fbb9a2fa17f2766a451cda0324c97b5d6a77e68f 2024-02-14T17:41:33.945 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:41:33 smithi017 systemd[1]: Started Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:34.644 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:34 smithi111 ceph-mon[25603]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:34.644 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:34.644 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:34 smithi017 ceph-mon[24769]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:34 smithi017 ceph-mon[29506]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:36.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:36 smithi111 ceph-mon[25603]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:36.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:36 smithi111 ceph-mon[25603]: Upgrade: Updating node-exporter.b (2/2) 2024-02-14T17:41:36.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:36 smithi111 ceph-mon[25603]: Deploying daemon node-exporter.b on smithi111 2024-02-14T17:41:36.485 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:36 smithi111 systemd[1]: Stopping Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:36 smithi017 ceph-mon[24769]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:36 smithi017 ceph-mon[24769]: Upgrade: Updating node-exporter.b (2/2) 2024-02-14T17:41:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:36 smithi017 ceph-mon[24769]: Deploying daemon node-exporter.b on smithi111 2024-02-14T17:41:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:36 smithi017 ceph-mon[29506]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:36 smithi017 ceph-mon[29506]: Upgrade: Updating node-exporter.b (2/2) 2024-02-14T17:41:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:36 smithi017 ceph-mon[29506]: Deploying daemon node-exporter.b on smithi111 2024-02-14T17:41:36.851 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:36 smithi111 podman[62660]: 2024-02-14 17:41:36.534552471 +0000 UTC m=+0.106167227 container died 5d05f9239e5aff5845bbcb9398dceadd3475555596b4619a47db0a8f6b4d9542 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:41:37.168 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:36 smithi111 podman[62660]: 2024-02-14 17:41:36.94937355 +0000 UTC m=+0.520988298 container cleanup 5d05f9239e5aff5845bbcb9398dceadd3475555596b4619a47db0a8f6b4d9542 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:41:37.168 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:36 smithi111 bash[62660]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b 2024-02-14T17:41:37.432 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:37 smithi111 podman[62674]: 2024-02-14 17:41:37.167309662 +0000 UTC m=+0.635679865 container remove 5d05f9239e5aff5845bbcb9398dceadd3475555596b4619a47db0a8f6b4d9542 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:41:37.432 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:37 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:41:37.432 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:37 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-14T17:41:37.432 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:37 smithi111 systemd[1]: Stopped Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:37.432 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:37 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Consumed 5.172s CPU time. 2024-02-14T17:41:37.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:37 smithi111 systemd[1]: Starting Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:38.234 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:37 smithi111 bash[62802]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-14T17:41:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:38 smithi017 ceph-mon[24769]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:41:38.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:38 smithi017 ceph-mon[29506]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:41:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:38 smithi111 ceph-mon[25603]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:41:38.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:38 smithi111 bash[62802]: Getting image source signatures 2024-02-14T17:41:38.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:38 smithi111 bash[62802]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-14T17:41:38.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:38 smithi111 bash[62802]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-14T17:41:38.735 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:38 smithi111 bash[62802]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-14T17:41:40.234 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:39 smithi111 bash[62802]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-14T17:41:40.234 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 bash[62802]: Writing manifest to image destination 2024-02-14T17:41:40.616 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-mon[25603]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:40.616 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 podman[62802]: 2024-02-14 17:41:40.314273805 +0000 UTC m=+2.611973837 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-14T17:41:40.616 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 podman[62802]: 2024-02-14 17:41:40.413081964 +0000 UTC m=+2.710781987 container create b2920c9965f16f1af7c8db068ede6929f67644f555ba4d5a5cf995a9f081bd72 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:41:40.616 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 podman[62802]: 2024-02-14 17:41:40.615371877 +0000 UTC m=+2.913071900 container init b2920c9965f16f1af7c8db068ede6929f67644f555ba4d5a5cf995a9f081bd72 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:41:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:40 smithi017 ceph-mon[24769]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:40 smithi017 ceph-mon[29506]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 podman[62802]: 2024-02-14 17:41:40.618173612 +0000 UTC m=+2.915873641 container start b2920c9965f16f1af7c8db068ede6929f67644f555ba4d5a5cf995a9f081bd72 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.622Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.622Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.622Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.622Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=arp 2024-02-14T17:41:40.985 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-14T17:41:40.986 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=edac 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-14T17:41:40.987 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=os 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=stat 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-14T17:41:40.988 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=time 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=uname 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b[62852]: ts=2024-02-14T17:41:40.623Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 bash[62802]: b2920c9965f16f1af7c8db068ede6929f67644f555ba4d5a5cf995a9f081bd72 2024-02-14T17:41:40.989 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:41:40 smithi111 systemd[1]: Started Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:41.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:41.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:41.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:41.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[24769]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:41.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:41:41] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-02-14T17:41:41.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:41.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:41.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:41.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:41 smithi017 ceph-mon[29506]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:41 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:41 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:41 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:41 smithi111 ceph-mon[25603]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:43.119 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:42 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:43.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:42 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:43.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:42 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:43.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:43 smithi111 ceph-mon[25603]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:41:44.068 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:43 smithi017 ceph-mon[24769]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:41:44.069 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:43 smithi017 ceph-mon[29506]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:41:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[24769]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[29506]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:45 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:46.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:45 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:45 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:45 smithi111 ceph-mon[25603]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:46.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:45 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:45 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:46.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:45 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:41:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:46 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:46 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:46 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:46 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:48 smithi017 ceph-mon[24769]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:41:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:48 smithi017 ceph-mon[29506]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:41:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:48 smithi111 ceph-mon[25603]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:41:50.418 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.419 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.420 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.420 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.420 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:50 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:50.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:50 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:41:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:51 smithi017 ceph-mon[24769]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-02-14T17:41:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:51 smithi017 ceph-mon[24769]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:51.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:51 smithi017 ceph-mon[29506]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-02-14T17:41:51.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:51 smithi017 ceph-mon[29506]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:51 smithi111 ceph-mon[25603]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-02-14T17:41:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:51 smithi111 ceph-mon[25603]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:41:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:41:51] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-02-14T17:41:52.466 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:52 smithi111 ceph-mon[25603]: Upgrade: Updating prometheus.a 2024-02-14T17:41:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:52 smithi017 ceph-mon[24769]: Upgrade: Updating prometheus.a 2024-02-14T17:41:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:52 smithi017 ceph-mon[29506]: Upgrade: Updating prometheus.a 2024-02-14T17:41:53.565 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:53 smithi111 ceph-mon[25603]: Deploying daemon prometheus.a on smithi111 2024-02-14T17:41:53.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:53 smithi111 ceph-mon[25603]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:53.566 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:53 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:53 smithi017 ceph-mon[29506]: Deploying daemon prometheus.a on smithi111 2024-02-14T17:41:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:53 smithi017 ceph-mon[29506]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:53 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:53 smithi017 ceph-mon[24769]: Deploying daemon prometheus.a on smithi111 2024-02-14T17:41:53.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:53 smithi017 ceph-mon[24769]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:53.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:53 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:41:55.272 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 systemd[1]: Stopping Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:55.555 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-mon[25603]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:55.555 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.270Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[60781]: ts=2024-02-14T17:41:55.271Z caller=main.go:1066 level=info msg="See you next time!" 2024-02-14T17:41:55.556 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 podman[64215]: 2024-02-14 17:41:55.281056628 +0000 UTC m=+0.131179606 container died 6143817423aafe5b51aebff66a81b4557617e055dd885c6bf122a316280db5ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:55.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:55 smithi017 ceph-mon[24769]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:55.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:55 smithi017 ceph-mon[29506]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:55.897 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 podman[64215]: 2024-02-14 17:41:55.645460019 +0000 UTC m=+0.495582998 container cleanup 6143817423aafe5b51aebff66a81b4557617e055dd885c6bf122a316280db5ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:55.897 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 bash[64215]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:41:56.197 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:55 smithi111 podman[64229]: 2024-02-14 17:41:55.896513507 +0000 UTC m=+0.617967305 container remove 6143817423aafe5b51aebff66a81b4557617e055dd885c6bf122a316280db5ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:56.198 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-14T17:41:56.198 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 systemd[1]: Stopped Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:56.198 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service: Consumed 1.191s CPU time. 2024-02-14T17:41:56.457 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 systemd[1]: Starting Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:41:56.457 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 podman[64361]: 2024-02-14 17:41:56.3135496 +0000 UTC m=+0.019154458 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-14T17:41:56.458 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 podman[64361]: 2024-02-14 17:41:56.456463118 +0000 UTC m=+0.162068032 container create 51b0c15c913d518f1ca2e9ebfb8a242ad10871088f356a53754c055665231c75 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:56.727 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 podman[64361]: 2024-02-14 17:41:56.700883648 +0000 UTC m=+0.406488504 container init 51b0c15c913d518f1ca2e9ebfb8a242ad10871088f356a53754c055665231c75 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:56.728 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 podman[64361]: 2024-02-14 17:41:56.703635767 +0000 UTC m=+0.409240627 container start 51b0c15c913d518f1ca2e9ebfb8a242ad10871088f356a53754c055665231c75 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:41:56.985 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.726Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-14T17:41:56.985 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.727Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-14T17:41:56.985 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.727Z caller=main.go:570 level=info host_details="(Linux 5.14.0-420.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 9 18:10:28 UTC 2024 x86_64 smithi111 (none))" 2024-02-14T17:41:56.985 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.727Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-14T17:41:56.985 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.727Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-14T17:41:56.985 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.729Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.729Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.730Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.730Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.732Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.733Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=713.052µs 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.733Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.741Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.754Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.761Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.764Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.766Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.767Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-02-14T17:41:56.986 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.768Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.768Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.154µs wal_replay_duration=34.548227ms wbl_replay_duration=83ns total_replay_duration=35.316913ms 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.770Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.770Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.770Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 bash[64361]: 51b0c15c913d518f1ca2e9ebfb8a242ad10871088f356a53754c055665231c75 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 systemd[1]: Started Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.789Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.386224ms db_storage=830ns remote_storage=1.105µs web_handler=397ns query_engine=538ns scrape=413.671µs scrape_sd=75.626µs notify=11.291µs notify_sd=13.696µs rules=17.645378ms tracing=7.511µs 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.789Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-14T17:41:56.987 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:41:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:41:56.789Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-14T17:41:57.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:57 smithi111 ceph-mon[25603]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-02-14T17:41:57.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:57 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:57.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:57 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:57.582 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:57 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[29506]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-02-14T17:41:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[24769]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-02-14T17:41:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:57.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:41:57.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:57 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:41:59.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:41:59 smithi111 ceph-mon[25603]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:41:59 smithi017 ceph-mon[24769]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:41:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:41:59 smithi017 ceph-mon[29506]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:42:01.376 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:42:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[24769]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:01.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:01.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[29506]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:01.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:01.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:01.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:01 smithi111 ceph-mon[25603]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (13m) 16s ago 21m 24.1M - 0.23.0 ba2b418f427c 556f84c8c0ad 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (13m) 1s ago 21m 56.5M - 8.3.5 dad864ee21e9 3af96da5f3f2 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 unknown 16s ago 20m 55.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (44s) 1s ago 26m 450M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (10m) 16s ago 27m 498M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (27m) 16s ago 27m 78.7M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (26m) 1s ago 26m 60.6M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:42:02.040 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (26m) 16s ago 26m 55.5M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (28s) 16s ago 22m 8036k - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (21s) 1s ago 21m 7860k - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (25m) 16s ago 25m 55.0M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (25m) 16s ago 25m 54.7M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (24m) 16s ago 24m 52.5M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (24m) 16s ago 24m 51.6M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (24m) 1s ago 24m 55.2M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (23m) 1s ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (23m) 1s ago 23m 51.5M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (23m) 1s ago 23m 56.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (5s) 1s ago 22m 46.1M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (20m) 16s ago 20m 77.6M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:42:02.041 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (20m) 1s ago 20m 77.9M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:42:02.441 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:42:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:42:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:42:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:02.441 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:42:02.442 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:42:02.594 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:02 smithi111 ceph-mon[25603]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:02.594 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:02 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:02.594 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:02 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:02 smithi017 ceph-mon[29506]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:02 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:02 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:02 smithi017 ceph-mon[24769]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:02 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:02 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:02.779 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:42:02.779 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:42:02.779 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:42:02.779 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-14T17:42:02.779 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-02-14T17:42:02.780 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-02-14T17:42:02.780 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:42:02.780 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/2 daemons upgraded", 2024-02-14T17:42:02.780 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading prometheus daemons", 2024-02-14T17:42:02.780 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:42:02.780 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:42:03.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:03 smithi111 ceph-mon[25603]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:03.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:03 smithi111 ceph-mon[25603]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:03.576 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:03 smithi111 ceph-mon[25603]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:03 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3611884007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:03.577 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:03 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[29506]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[29506]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[29506]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3611884007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:03.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[24769]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:03.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[24769]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:03.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[24769]: from='client.25432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:03.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3611884007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:03.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:03 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:04.641 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:04 smithi111 ceph-mon[25603]: from='client.25444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:04 smithi017 ceph-mon[24769]: from='client.25444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:04 smithi017 ceph-mon[29506]: from='client.25444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:05 smithi017 ceph-mon[24769]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:05 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:05 smithi017 ceph-mon[29506]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:05 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:05 smithi111 ceph-mon[25603]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:05 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:06.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:06 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:06.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:06 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:08 smithi017 ceph-mon[24769]: Upgrade: Updating alertmanager.a 2024-02-14T17:42:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:08 smithi017 ceph-mon[24769]: Deploying daemon alertmanager.a on smithi017 2024-02-14T17:42:08.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:08 smithi017 ceph-mon[24769]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:08 smithi017 ceph-mon[29506]: Upgrade: Updating alertmanager.a 2024-02-14T17:42:08.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:08 smithi017 ceph-mon[29506]: Deploying daemon alertmanager.a on smithi017 2024-02-14T17:42:08.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:08 smithi017 ceph-mon[29506]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:08 smithi111 ceph-mon[25603]: Upgrade: Updating alertmanager.a 2024-02-14T17:42:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:08 smithi111 ceph-mon[25603]: Deploying daemon alertmanager.a on smithi017 2024-02-14T17:42:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:08 smithi111 ceph-mon[25603]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:10 smithi017 ceph-mon[24769]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:10 smithi017 ceph-mon[29506]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:10.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:10 smithi017 systemd[1]: Stopping Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:42:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:10 smithi111 ceph-mon[25603]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:11.190 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[59165]: level=info ts=2024-02-14T17:42:10.900Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:42:11.190 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:10 smithi017 podman[78500]: 2024-02-14 17:42:10.971383092 +0000 UTC m=+0.204118510 container died 556f84c8c0ade76aaa9fbd0293dfce888636e013f6ce9476edef8c9ed0da0aff (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:11.611 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:11 smithi017 podman[78500]: 2024-02-14 17:42:11.350247296 +0000 UTC m=+0.582982722 container cleanup 556f84c8c0ade76aaa9fbd0293dfce888636e013f6ce9476edef8c9ed0da0aff (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:11.611 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:11 smithi017 bash[78500]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a 2024-02-14T17:42:11.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:42:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:42:11] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.43.0" 2024-02-14T17:42:11.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:11 smithi017 podman[78515]: 2024-02-14 17:42:11.610173674 +0000 UTC m=+0.635992966 container remove 556f84c8c0ade76aaa9fbd0293dfce888636e013f6ce9476edef8c9ed0da0aff (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:11.941 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:11 smithi017 podman[78515]: 2024-02-14 17:42:11.694311157 +0000 UTC m=+0.720130420 volume remove e46bf647d1f4cadffc7bca027e6cc57fc01bf773aa2596e37be38401dd08966d 2024-02-14T17:42:11.941 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:11 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-14T17:42:11.941 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:11 smithi017 systemd[1]: Stopped Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:42:11.941 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:11 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service: Consumed 1.478s CPU time. 2024-02-14T17:42:12.394 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 systemd[1]: Starting Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:42:12.394 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 podman[78640]: 2024-02-14 17:42:12.195478918 +0000 UTC m=+0.091819609 volume create 8b8e97e1e6197a0da2fe63f7d0b6ff67d5ef27a3ff3b36c852305ea7adf36ac4 2024-02-14T17:42:12.394 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 podman[78640]: 2024-02-14 17:42:12.121798796 +0000 UTC m=+0.018139503 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-14T17:42:12.394 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 podman[78640]: 2024-02-14 17:42:12.254198811 +0000 UTC m=+0.150539496 container create 3ba623745a7f49969c82265b7e36e9b5edf28612a2e4c0611763db503416f0c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:12.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-mon[24769]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-02-14T17:42:12.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-mon[29506]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-02-14T17:42:12.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 podman[78640]: 2024-02-14 17:42:12.54861839 +0000 UTC m=+0.444959079 container init 3ba623745a7f49969c82265b7e36e9b5edf28612a2e4c0611763db503416f0c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:12.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 podman[78640]: 2024-02-14 17:42:12.551378872 +0000 UTC m=+0.447719561 container start 3ba623745a7f49969c82265b7e36e9b5edf28612a2e4c0611763db503416f0c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.563Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.563Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.564Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.17 port=9094 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.565Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.588Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.589Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.592Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:12.592Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 bash[78640]: 3ba623745a7f49969c82265b7e36e9b5edf28612a2e4c0611763db503416f0c7 2024-02-14T17:42:12.691 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:12 smithi017 systemd[1]: Started Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:42:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:12 smithi111 ceph-mon[25603]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:13.675 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:13 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:13 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:13 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:13 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:13 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:14.567 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:14 smithi017 ceph-mon[24769]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:14.568 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:14 smithi017 ceph-mon[29506]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:14 smithi111 ceph-mon[25603]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:42:14.939 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:14.566Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000593158s 2024-02-14T17:42:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:16 smithi017 ceph-mon[29506]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:16 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:16 smithi017 ceph-mon[24769]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:16 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:16 smithi111 ceph-mon[25603]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:16 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:17.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:17 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:17.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:17 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:17.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:17 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:17.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:17 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:17.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:17 smithi111 ceph-mon[25603]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[24769]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:18.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:17 smithi017 ceph-mon[29506]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:19.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:19 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:19.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:19 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:19 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:19.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:19 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:19.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:19 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:19.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:19 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:20.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:20 smithi017 ceph-mon[24769]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:20.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:20 smithi017 ceph-mon[29506]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:20 smithi111 ceph-mon[25603]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:42:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:42:21] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.43.0" 2024-02-14T17:42:22.656 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:22.656 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.656 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.656 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:22.656 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.657 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.658 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.658 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:22 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:42:22.568Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002214773s 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:22.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:22.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:23.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[24769]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:23.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[24769]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:23.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[24769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:42:23.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:23.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[29506]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:23.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[29506]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:23.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[29506]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:42:23.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:23 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:23 smithi111 ceph-mon[25603]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:23.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:23 smithi111 ceph-mon[25603]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:23.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:23 smithi111 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-14T17:42:23.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:23 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:25.543 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:25 smithi111 ceph-mon[25603]: Upgrade: Updating grafana.a 2024-02-14T17:42:25.543 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:25 smithi111 ceph-mon[25603]: Deploying daemon grafana.a on smithi111 2024-02-14T17:42:25.543 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:25 smithi111 ceph-mon[25603]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:25.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:25 smithi017 ceph-mon[24769]: Upgrade: Updating grafana.a 2024-02-14T17:42:25.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:25 smithi017 ceph-mon[24769]: Deploying daemon grafana.a on smithi111 2024-02-14T17:42:25.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:25 smithi017 ceph-mon[24769]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:25.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:25 smithi017 ceph-mon[29506]: Upgrade: Updating grafana.a 2024-02-14T17:42:25.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:25 smithi017 ceph-mon[29506]: Deploying daemon grafana.a on smithi111 2024-02-14T17:42:25.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:25 smithi017 ceph-mon[29506]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:26.845 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:26 smithi111 systemd[1]: Stopping Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:42:26.845 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[49731]: t=2024-02-14T17:42:26+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-02-14T17:42:27.102 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:26 smithi111 podman[66324]: 2024-02-14 17:42:26.844377243 +0000 UTC m=+0.106396879 container died 3af96da5f3f24efcbd60c650cc0952ade4b3f37f528715a8d07f7db04a46b898 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, distribution-scope=public, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , name=ubi8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, architecture=x86_64) 2024-02-14T17:42:27.364 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 podman[66324]: 2024-02-14 17:42:27.175632009 +0000 UTC m=+0.437651654 container cleanup 3af96da5f3f24efcbd60c650cc0952ade4b3f37f528715a8d07f7db04a46b898 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, name=ubi8, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.openshift.expose-services=, io.openshift.tags=base rhel8, version=8.5, vcs-type=git, description=Ceph Grafana Container, com.redhat.component=ubi8-container) 2024-02-14T17:42:27.364 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 bash[66324]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a 2024-02-14T17:42:27.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:27 smithi017 ceph-mon[24769]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-02-14T17:42:27.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:27 smithi017 ceph-mon[29506]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-02-14T17:42:27.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:27 smithi111 ceph-mon[25603]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-02-14T17:42:27.734 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 podman[66338]: 2024-02-14 17:42:27.360244231 +0000 UTC m=+0.517687686 container remove 3af96da5f3f24efcbd60c650cc0952ade4b3f37f528715a8d07f7db04a46b898 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, com.redhat.component=ubi8-container, version=8.5, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=Ceph Grafana Container, io.openshift.tags=base rhel8, architecture=x86_64, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-14T17:42:27.734 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service: Deactivated successfully. 2024-02-14T17:42:27.734 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 systemd[1]: Stopped Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:42:27.735 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service: Consumed 5.241s CPU time. 2024-02-14T17:42:28.093 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 systemd[1]: Starting Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:42:28.093 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 podman[66471]: 2024-02-14 17:42:27.880270897 +0000 UTC m=+0.016849490 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-14T17:42:28.094 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:27 smithi111 podman[66471]: 2024-02-14 17:42:27.995268215 +0000 UTC m=+0.131846852 container create 7b859cc99d088a9e3fd12227c65865b821610bd61018f692067dde307d3d4e02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vcs-type=git, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, maintainer=Paul Cuzner , release=239.1651231664, io.openshift.tags=base rhel8, distribution-scope=public, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.openshift.expose-services=) 2024-02-14T17:42:28.359 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 podman[66471]: 2024-02-14 17:42:28.205963715 +0000 UTC m=+0.342542307 container init 7b859cc99d088a9e3fd12227c65865b821610bd61018f692067dde307d3d4e02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, vcs-type=git, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, release=239.1651231664, com.redhat.component=ubi8-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , version=8.5) 2024-02-14T17:42:28.359 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 podman[66471]: 2024-02-14 17:42:28.208693742 +0000 UTC m=+0.345272335 container start 7b859cc99d088a9e3fd12227c65865b821610bd61018f692067dde307d3d4e02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, release=239.1651231664, version=8.5, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-02-14T17:42:28.359 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 bash[66471]: 7b859cc99d088a9e3fd12227c65865b821610bd61018f692067dde307d3d4e02 2024-02-14T17:42:28.360 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 systemd[1]: Started Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:42:28.625 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358569813Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-14T17:42:28.625 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.35878567Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358816396Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358829071Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358839556Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358848636Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358854406Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358860369Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358866195Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.35887179Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358877239Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358882556Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.358888265Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-14T17:42:28.626 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=settings t=2024-02-14T17:42:28.35889395Z level=info msg="App mode production" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=sqlstore t=2024-02-14T17:42:28.358942495Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=sqlstore t=2024-02-14T17:42:28.358959018Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.370155627Z level=info msg="Starting DB migrations" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.373655208Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.383367045Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.384637284Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.385421466Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.386387115Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.387387879Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.388351338Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.389316585Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.390220103Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.390995883Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-14T17:42:28.627 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.391341966Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.392345697Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.393220105Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.39421414Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.39515099Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.396093907Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.398901101Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.401657014Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.402661878Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-14T17:42:28.628 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.405484554Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-14T17:42:28.629 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.408290941Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-14T17:42:28.629 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.411011947Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-14T17:42:28.629 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.411473006Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-14T17:42:28.629 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.41428015Z level=info msg="Executing migration" id="create alert_image table" 2024-02-14T17:42:28.629 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.416935675Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-14T17:42:28.629 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.419836918Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-14T17:42:28.629 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.420166855Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.422932997Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.425848578Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.425933908Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.428609821Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.429160214Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.431912056Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.434683163Z level=info msg="Executing migration" id="create secrets table" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.437731726Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.44261321Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.445343199Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.448049267Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.452754472Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.457520032Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.460165723Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-14T17:42:28.630 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.462939536Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.465641956Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.465975509Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.468505968Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.469834634Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.471251341Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.473858779Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.47421505Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.476922671Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.479649828Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.482299592Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.48305618Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.4838872Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.484745273Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.485566742Z level=info msg="Executing migration" id="add correlation config column" 2024-02-14T17:42:28.631 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.486554103Z level=info msg="Executing migration" id="create entity_events table" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.48740844Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.488235039Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.488340325Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.48907065Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.489165386Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.48957805Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.490425419Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.491201105Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.492056769Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.49253255Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.493379167Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.494192779Z level=info msg="Executing migration" id="Drop public config table" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.495060342Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-14T17:42:28.632 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.495504906Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.496348471Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.497192629Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.498089504Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.501157831Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.502228152Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.50328585Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.504130823Z level=info msg="Executing migration" id="add share column" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.505244253Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.506059909Z level=info msg="Executing migration" id="create file table" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.506943025Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.507761392Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.508232739Z level=info msg="Executing migration" id="create file_meta table" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.50912261Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.509941745Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.510800559Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-14T17:42:28.633 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.511911873Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.512768542Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.513650892Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.516361538Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.519005949Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.521798878Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.524588661Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.527268442Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.52767111Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.530459988Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.533175902Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.535868957Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.545702718Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.548385819Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-14T17:42:28.634 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.551227343Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.554655215Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.555218819Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.55801468Z level=info msg="Executing migration" id="create folder table" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.558428533Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.560789065Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.563456868Z level=info msg="Executing migration" id="Update folder title length" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.563787095Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=migrator t=2024-02-14T17:42:28.566456082Z level=info msg="migrations completed" performed=105 skipped=376 duration=192.848179ms 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=sqlstore t=2024-02-14T17:42:28.56680609Z level=info msg="Created default organization" 2024-02-14T17:42:28.635 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=plugin.loader t=2024-02-14T17:42:28.624199237Z level=info msg="Plugin registered" pluginID=input 2024-02-14T17:42:28.951 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=plugin.loader t=2024-02-14T17:42:28.652897741Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-14T17:42:28.951 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=plugin.loader t=2024-02-14T17:42:28.652925767Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-14T17:42:28.951 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=secrets t=2024-02-14T17:42:28.652964805Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-14T17:42:28.951 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=query_data t=2024-02-14T17:42:28.655189744Z level=info msg="Query Service initialization" 2024-02-14T17:42:28.951 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=live.push_http t=2024-02-14T17:42:28.657879458Z level=info msg="Live Push Gateway initialization" 2024-02-14T17:42:29.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-mon[25603]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:29.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:29.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:29.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=infra.usagestats.collector t=2024-02-14T17:42:29.298956388Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=provisioning.datasources t=2024-02-14T17:42:29.301624734Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=provisioning.datasources t=2024-02-14T17:42:29.301649117Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=provisioning.datasources t=2024-02-14T17:42:29.301769021Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=provisioning.alerting t=2024-02-14T17:42:29.329726259Z level=info msg="starting to provision alerting" 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=provisioning.alerting t=2024-02-14T17:42:29.32996215Z level=info msg="finished to provision alerting" 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=ngalert.state.manager t=2024-02-14T17:42:29.330167277Z level=info msg="Warming state cache for startup" 2024-02-14T17:42:29.486 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=grafanaStorageLogger t=2024-02-14T17:42:29.330402088Z level=info msg="storage starting" 2024-02-14T17:42:29.487 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=http.server t=2024-02-14T17:42:29.332531551Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-14T17:42:29.487 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=ngalert.state.manager t=2024-02-14T17:42:29.340054272Z level=info msg="State cache has been initialized" states=0 duration=9.884526ms 2024-02-14T17:42:29.487 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=ticker t=2024-02-14T17:42:29.340376475Z level=info msg=starting first_tick=2024-02-14T17:42:30Z 2024-02-14T17:42:29.487 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=ngalert.multiorg.alertmanager t=2024-02-14T17:42:29.340566771Z level=info msg="starting MultiOrg Alertmanager" 2024-02-14T17:42:29.487 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=sqlstore.transactions t=2024-02-14T17:42:29.362813521Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-14T17:42:29.487 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:42:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=sqlstore.transactions t=2024-02-14T17:42:29.374391832Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-14T17:42:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[24769]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[29506]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:29.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:29.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:29 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:31.638 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:31 smithi111 ceph-mon[25603]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:31.638 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:31 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:31 smithi017 ceph-mon[24769]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:31 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:31 smithi017 ceph-mon[29506]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:31 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:32.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:42:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:42:31] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.43.0" 2024-02-14T17:42:33.135 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[29506]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[24769]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:33.393 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:33 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:33 smithi111 ceph-mon[25603]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:33 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:33 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:33 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:33.815 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:42:33.815 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (21s) 16s ago 22m 15.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:42:33.815 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (5s) 1s ago 21m 52.0M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 unknown 16s ago 21m 58.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (76s) 1s ago 26m 451M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (11m) 16s ago 28m 504M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (28m) 16s ago 28m 79.1M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (26m) 1s ago 26m 61.3M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (26m) 16s ago 26m 56.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (60s) 16s ago 22m 7914k - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (53s) 1s ago 22m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (26m) 16s ago 26m 55.0M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (25m) 16s ago 25m 54.7M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (25m) 16s ago 25m 52.6M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (25m) 16s ago 25m 51.6M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (24m) 1s ago 24m 55.2M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:42:33.816 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (24m) 1s ago 24m 55.3M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:42:33.817 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (24m) 1s ago 24m 51.5M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:42:33.817 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (23m) 1s ago 23m 56.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:42:33.817 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (37s) 1s ago 22m 49.0M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:42:33.817 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (21m) 16s ago 21m 77.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:42:33.817 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (21m) 1s ago 21m 78.0M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:42:34.554 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:42:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:42:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:42:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-14T17:42:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-02-14T17:42:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr" 2024-02-14T17:42:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:42:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/2 daemons upgraded", 2024-02-14T17:42:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading grafana daemons", 2024-02-14T17:42:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:42:34.555 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:42:34.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[24769]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:34.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[24769]: from='client.25456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:34.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:34.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:34.555 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2287130516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:34.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[29506]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:34.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[29506]: from='client.25456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:34.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:34.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:34.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:34 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2287130516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:34.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:34 smithi111 ceph-mon[25603]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:34.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:34 smithi111 ceph-mon[25603]: from='client.25456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:34.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:34.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:34.655 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:34 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2287130516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:35.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:35 smithi111 ceph-mon[25603]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:35.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:35 smithi111 ceph-mon[25603]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:35.578 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:35 smithi111 ceph-mon[25603]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:35 smithi017 ceph-mon[24769]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:35 smithi017 ceph-mon[24769]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:35 smithi017 ceph-mon[24769]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:35 smithi017 ceph-mon[29506]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:35 smithi017 ceph-mon[29506]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:35 smithi017 ceph-mon[29506]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:42:37.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:37.368 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.369 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:42:37.371 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:42:37.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:42:37.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:42:37.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:42:37.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:42:37.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:42:37.372 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:42:37.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:42:37.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:42:37.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:42:37.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:42:37.373 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:42:37.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:42:37.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:42:37.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:42:37.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:42:37.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:42:37.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.374 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.375 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.376 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:37.376 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.376 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.376 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.376 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.376 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:42:37.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.378 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.378 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.378 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.378 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.378 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.378 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.379 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.379 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.379 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.379 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.379 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:42:37.379 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:42:37.380 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.380 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:42:37.380 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:42:37.380 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.381 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.382 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.382 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:42:37.382 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:42:37.382 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:42:37.382 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:42:37.382 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:42:37.382 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.383 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.384 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.387 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:37 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:42:37.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:37.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:42:37.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:42:37.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:42:37.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:37 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:42:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:42:38.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: Upgrade: Finalizing container_image settings 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: Upgrade: Complete! 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[24769]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: Upgrade: Finalizing container_image settings 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: Upgrade: Complete! 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:38.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-02-14T17:42:38.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-14T17:42:38.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:38 smithi017 ceph-mon[29506]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-02-14T17:42:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:42:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-14T17:42:38.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: Upgrade: Finalizing container_image settings 2024-02-14T17:42:38.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: Upgrade: Complete! 2024-02-14T17:42:38.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:42:38.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-02-14T17:42:38.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-14T17:42:38.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:38 smithi111 ceph-mon[25603]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 3 op/s 2024-02-14T17:42:40.628 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:40 smithi111 ceph-mon[25603]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:42:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:40 smithi017 ceph-mon[24769]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:42:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:40 smithi017 ceph-mon[29506]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:42:41.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:41 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:41.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:41 smithi017 ceph-mon[24769]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:42:41.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:41 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:41.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:41 smithi017 ceph-mon[29506]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:42:41.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:42:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:42:41] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.43.0" 2024-02-14T17:42:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:41 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:42:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:41 smithi111 ceph-mon[25603]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:42:42.863 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:42 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:42.863 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:42 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:42 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:43.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:43 smithi017 ceph-mon[24769]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-14T17:42:43.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:43 smithi017 ceph-mon[29506]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-14T17:42:43.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:43 smithi111 ceph-mon[25603]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 842 B/s rd, 0 op/s 2024-02-14T17:42:46.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:46 smithi111 ceph-mon[25603]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-02-14T17:42:46.688 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:46 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:46.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:46 smithi017 ceph-mon[29506]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-02-14T17:42:46.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:46 smithi017 ceph-mon[24769]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s 2024-02-14T17:42:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:42:48.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:48 smithi111 ceph-mon[25603]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:48 smithi017 ceph-mon[24769]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:48 smithi017 ceph-mon[29506]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:42:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:50 smithi017 ceph-mon[24769]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:50 smithi017 ceph-mon[29506]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:50 smithi111 ceph-mon[25603]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:42:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:42:51] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.43.0" 2024-02-14T17:42:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:52 smithi017 ceph-mon[24769]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:52 smithi017 ceph-mon[29506]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:52 smithi111 ceph-mon[25603]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:53 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:53 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:53 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:42:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:54 smithi017 ceph-mon[24769]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:54 smithi017 ceph-mon[29506]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:54 smithi111 ceph-mon[25603]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:56 smithi017 ceph-mon[24769]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:56 smithi017 ceph-mon[29506]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:56 smithi111 ceph-mon[25603]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:42:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:42:58 smithi017 ceph-mon[24769]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:42:58 smithi017 ceph-mon[29506]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:42:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:42:58 smithi111 ceph-mon[25603]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:00 smithi017 ceph-mon[24769]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:00 smithi017 ceph-mon[29506]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:00 smithi111 ceph-mon[25603]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:01 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:43:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:43:01] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.43.0" 2024-02-14T17:43:02.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:02 smithi017 ceph-mon[24769]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:02.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:02 smithi017 ceph-mon[29506]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:02 smithi111 ceph-mon[25603]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:03.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:03 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:03 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:03 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:04.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:04 smithi017 ceph-mon[24769]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:04 smithi017 ceph-mon[29506]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:04 smithi111 ceph-mon[25603]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:05.588 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-14T17:43:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:06 smithi017 ceph-mon[24769]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:06 smithi017 ceph-mon[24769]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:06 smithi017 ceph-mon[29506]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:06 smithi017 ceph-mon[29506]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:06 smithi111 ceph-mon[25603]: from='client.15567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:06 smithi111 ceph-mon[25603]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:07.708 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:43:08.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:08 smithi017 ceph-mon[24769]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:08.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:08 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2133357596' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:08.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:08 smithi017 ceph-mon[29506]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:08.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:08 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2133357596' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:08.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-14T17:43:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:08 smithi111 ceph-mon[25603]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:08.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:08 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2133357596' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:10.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:10 smithi017 ceph-mon[24769]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:10.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:10 smithi017 ceph-mon[29506]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:10.718 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)" 2024-02-14T17:43:10.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:10 smithi111 ceph-mon[25603]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:11.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-02-14T17:43:11.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:11 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3559911948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:11.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:11 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3559911948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:11.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:11 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3559911948' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:11.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:43:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:43:11] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.43.0" 2024-02-14T17:43:12.622 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:12 smithi017 ceph-mon[29506]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:12.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:12 smithi017 ceph-mon[24769]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:12.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:12 smithi111 ceph-mon[25603]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:13.422 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:43:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:13 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:13 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:13.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:13 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:14.028 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-02-14T17:43:14.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:14 smithi017 ceph-mon[24769]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:14.633 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:14 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3306526842' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:14.633 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:14 smithi017 ceph-mon[29506]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:14.633 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:14 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3306526842' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:14 smithi111 ceph-mon[25603]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:14 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/3306526842' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:16 smithi017 ceph-mon[29506]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:16 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:16 smithi017 ceph-mon[24769]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:16 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:16 smithi111 ceph-mon[25603]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:16 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:17.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:17 smithi017 ceph-mon[29506]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:17.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:17 smithi017 ceph-mon[24769]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:17.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:17 smithi111 ceph-mon[25603]: from='client.15591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:18.090 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:43:18.636 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:18 smithi017 ceph-mon[24769]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:18.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:18 smithi017 ceph-mon[29506]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:18.664 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:43:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:18 smithi111 ceph-mon[25603]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:20.564 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:20 smithi017 ceph-mon[24769]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:20.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:20 smithi017 ceph-mon[29506]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:20.682 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:43:20.682 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": null, 2024-02-14T17:43:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": false, 2024-02-14T17:43:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "", 2024-02-14T17:43:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:43:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": null, 2024-02-14T17:43:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-02-14T17:43:20.683 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:43:20.683 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:43:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:20 smithi111 ceph-mon[25603]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:21.275 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:43:21.870 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:43:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:43:21] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.43.0" 2024-02-14T17:43:22.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:22 smithi017 ceph-mon[24769]: from='client.25510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:22.543 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:22 smithi017 ceph-mon[24769]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:22.544 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:22 smithi017 ceph-mon[29506]: from='client.25510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:22.544 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:22 smithi017 ceph-mon[29506]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:22 smithi111 ceph-mon[25603]: from='client.25510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:22 smithi111 ceph-mon[25603]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:23.249 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:43:23.249 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:43:23.250 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:43:23.619 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:23 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:23.620 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:23 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2567163434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:43:23.620 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:23 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:23.620 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:23 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2567163434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:43:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:23 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:23 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/2567163434' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:43:23.898 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-02-14T17:43:24.493 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:24 smithi017 ceph-mon[24769]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:24.493 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:24 smithi017 ceph-mon[29506]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:24 smithi111 ceph-mon[25603]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:26.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:26 smithi017 ceph-mon[24769]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:26 smithi017 ceph-mon[29506]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:26 smithi111 ceph-mon[25603]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:27.399 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:27 smithi017 ceph-mon[24769]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:27.400 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:27 smithi017 ceph-mon[24769]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mon", "hosts": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:27.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:27 smithi017 ceph-mon[29506]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:27.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:27 smithi017 ceph-mon[29506]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mon", "hosts": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:27.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:27 smithi111 ceph-mon[25603]: from='client.15609 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:27.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:27 smithi111 ceph-mon[25603]: from='client.15615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mon", "hosts": "smithi111", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:28.196 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:43:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:28 smithi017 ceph-mon[29506]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:28 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:28 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:28 smithi017 ceph-mon[24769]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:28 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:28 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:28 smithi111 ceph-mon[25603]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:28 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:28 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:28.831 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-14T17:43:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[24769]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:43:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:29.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[29506]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:43:29.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:29.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:29.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:29 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:29.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:29 smithi111 ceph-mon[25603]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:43:29.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:29 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:29.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:29 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:29.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:29 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:30.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:30 smithi017 ceph-mon[24769]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-14T17:43:30.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:30 smithi017 ceph-mon[24769]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:30.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:30 smithi017 ceph-mon[24769]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:43:30.544 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:30 smithi017 ceph-mon[29506]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-14T17:43:30.544 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:30 smithi017 ceph-mon[29506]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:30.544 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:30 smithi017 ceph-mon[29506]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:43:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:30 smithi111 ceph-mon[25603]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-14T17:43:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:30 smithi111 ceph-mon[25603]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:30 smithi111 ceph-mon[25603]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:43:31.009 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:43:31.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:31 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:31.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:31 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (79s) 74s ago 23m 15.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (63s) 59s ago 22m 52.0M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 unknown 74s ago 22m 58.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (2m) 59s ago 27m 451M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (12m) 74s ago 29m 504M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (29m) 74s ago 29m 79.1M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (27m) 59s ago 27m 61.3M 2048M 17.2.0 e1d6a67b021e 4257ce96e8cf 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (27m) 74s ago 27m 56.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (118s) 74s ago 23m 7914k - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (111s) 59s ago 23m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:43:31.684 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (27m) 74s ago 27m 55.0M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (26m) 74s ago 26m 54.7M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (26m) 74s ago 26m 52.6M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (26m) 74s ago 26m 51.6M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (25m) 59s ago 25m 55.2M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (25m) 59s ago 25m 55.3M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (24m) 59s ago 24m 51.5M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (24m) 59s ago 24m 56.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (95s) 59s ago 23m 49.0M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (22m) 74s ago 22m 77.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:43:31.685 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (22m) 59s ago 22m 78.0M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:43:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:31 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:32.089 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:43:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:43:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:43:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:43:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:43:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-02-14T17:43:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:43:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:43:32.091 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:43:32.091 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:43:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:43:31] "GET /metrics HTTP/1.1" 200 37989 "" "Prometheus/2.43.0" 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi111", 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "0/1 daemons upgraded", 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mon daemons", 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:43:32.428 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:43:32.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:32.662 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-14T17:43:32.663 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:32 smithi111 ceph-mon[25603]: from='client.? 172.21.15.17:0/177250744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:32.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/177250744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:43:32.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-02-14T17:43:32.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:32 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/177250744' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:33 smithi111 ceph-mon[25603]: from='client.25546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:33 smithi111 ceph-mon[25603]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:43:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:33 smithi111 ceph-mon[25603]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:43:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:33 smithi111 ceph-mon[25603]: Upgrade: Setting container_image for all mgr 2024-02-14T17:43:33.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:33 smithi111 ceph-mon[25603]: Upgrade: It appears safe to stop mon.b 2024-02-14T17:43:33.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:33 smithi111 ceph-mon[25603]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:33.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:33 smithi111 ceph-mon[25603]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[24769]: from='client.25546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[24769]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:43:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[24769]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:43:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all mgr 2024-02-14T17:43:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[24769]: Upgrade: It appears safe to stop mon.b 2024-02-14T17:43:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[24769]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:33.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[29506]: from='client.25546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:33.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[29506]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:43:33.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[29506]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:43:33.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all mgr 2024-02-14T17:43:33.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[29506]: Upgrade: It appears safe to stop mon.b 2024-02-14T17:43:33.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[29506]: from='client.15645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:43:33.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:33 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:34.426 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:34 smithi111 ceph-mon[25603]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:34.427 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:34.427 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:43:34.427 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:43:34.427 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:34 smithi111 ceph-mon[25603]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[24769]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:43:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:43:34.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[29506]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:43:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:43:34.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:34 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:35.422 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 systemd[1]: Stopping Ceph mon.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:43:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:35 smithi017 ceph-mon[24769]: Upgrade: Updating mon.b 2024-02-14T17:43:35.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:35 smithi017 ceph-mon[24769]: Deploying daemon mon.b on smithi111 2024-02-14T17:43:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:35 smithi017 ceph-mon[29506]: Upgrade: Updating mon.b 2024-02-14T17:43:35.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:35 smithi017 ceph-mon[29506]: Deploying daemon mon.b on smithi111 2024-02-14T17:43:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 ceph-mon[25603]: Upgrade: Updating mon.b 2024-02-14T17:43:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 ceph-mon[25603]: Deploying daemon mon.b on smithi111 2024-02-14T17:43:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b[25584]: 2024-02-14T17:43:35.538+0000 7f96573dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:43:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b[25584]: 2024-02-14T17:43:35.538+0000 7f96573dd700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-14T17:43:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 podman[67931]: 2024-02-14 17:43:35.580172111 +0000 UTC m=+0.133306846 container died 4257ce96e8cf94dda8fe5bc7eb7d7fb511525e72ceab6ecc88784502cb8a0134 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., distribution-scope=public, architecture=x86_64, vcs-type=git, ceph=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0) 2024-02-14T17:43:36.216 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 podman[67931]: 2024-02-14 17:43:35.963796909 +0000 UTC m=+0.516931643 container cleanup 4257ce96e8cf94dda8fe5bc7eb7d7fb511525e72ceab6ecc88784502cb8a0134 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, distribution-scope=public, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, ceph=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, version=8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD) 2024-02-14T17:43:36.216 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:35 smithi111 bash[67931]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b 2024-02-14T17:43:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 podman[67945]: 2024-02-14 17:43:36.215323452 +0000 UTC m=+0.638182632 container remove 4257ce96e8cf94dda8fe5bc7eb7d7fb511525e72ceab6ecc88784502cb8a0134 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-type=git, RELEASE=HEAD, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, name=centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, version=8) 2024-02-14T17:43:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.b.service: Deactivated successfully. 2024-02-14T17:43:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 systemd[1]: Stopped Ceph mon.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:43:36.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.b.service: Consumed 18.296s CPU time. 2024-02-14T17:43:36.848 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 systemd[1]: Starting Ceph mon.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:43:36.848 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 podman[68079]: 2024-02-14 17:43:36.635557939 +0000 UTC m=+0.018062009 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:43:36.848 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 podman[68079]: 2024-02-14 17:43:36.741130968 +0000 UTC m=+0.123635032 container create bb055293bff4f5e3fce301838f5eb64ce0f1016bc42d6ebbb61328fbe4065665 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, ceph=True, org.label-schema.schema-version=1.0) 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 podman[68079]: 2024-02-14 17:43:36.943805021 +0000 UTC m=+0.326309091 container init bb055293bff4f5e3fce301838f5eb64ce0f1016bc42d6ebbb61328fbe4065665 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 podman[68079]: 2024-02-14 17:43:36.946960985 +0000 UTC m=+0.329465058 container start bb055293bff4f5e3fce301838f5eb64ce0f1016bc42d6ebbb61328fbe4065665 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: set uid:gid to 167:167 (ceph:ceph) 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev), process ceph-mon, pid 2 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: pidfile_write: ignore empty --pid-file 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: load: jerasure load: lrc 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: RocksDB version: 7.9.2 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Git sha 0 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Compile date 2024-02-14 11:59:27 2024-02-14T17:43:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: DB SUMMARY 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: DB Session ID: 3Q4W5E4UFNMJ922JAM41 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2814 Bytes 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2085433 ; 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.env: 0x559176157ae0 2024-02-14T17:43:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.fs: PosixFileSystem 2024-02-14T17:43:37.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.info_log: 0x55917710b8c0 2024-02-14T17:43:37.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:43:37.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.statistics: (nil) 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.use_fsync: 0 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:43:37.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.db_log_dir: 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.wal_dir: 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:43:37.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.write_buffer_manager: 0x5591770dedc0 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.unordered_write: 0 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:43:37.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.row_cache: None 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.wal_filter: None 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.wal_compression: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:43:37.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:43:37.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_open_files: -1 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Compression algorithms supported: 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kZSTD supported: 0 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:43:37.243 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.merge_operator: 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_filter: None 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55917710b9c0) 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:43:37.244 INFO:journalctl@ceph.mon.b.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: index_type: 0 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: data_block_index_type: 0 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: index_shortening: 1 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: checksum: 4 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: no_block_cache: 0 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache: 0x55917710f090 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache_options: 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: capacity : 536870912 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: num_shard_bits : 4 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: strict_capacity_limit : 0 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_cache_compressed: (nil) 2024-02-14T17:43:37.245 INFO:journalctl@ceph.mon.b.smithi111.stdout: persistent_cache: (nil) 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_size: 4096 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_size_deviation: 10 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_restart_interval: 16 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: index_block_restart_interval: 1 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: metadata_block_size: 4096 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: partition_filters: 0 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: use_delta_encoding: 1 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: filter_policy: bloomfilter 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: whole_key_filtering: 1 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: verify_compression: 0 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: format_version: 5 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: enable_index_compression: 1 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: block_align: 0 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: max_auto_readahead_size: 262144 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: prepopulate_block_cache: 0 2024-02-14T17:43:37.246 INFO:journalctl@ceph.mon.b.smithi111.stdout: initial_auto_readahead_size: 8192 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout: num_file_reads_for_auto_readahead: 2 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression: NoCompression 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.num_levels: 7 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:43:37.247 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:43:37.248 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:43:37.249 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.arena_block_size: 1048576 2024-02-14T17:43:37.250 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:43:37.251 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.ttl: 2592000 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:43:37.252 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.blob_file_starting_level: 0 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28876, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8d1e4347-68c9-4caa-8dac-e50739e818cb 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932616976344, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932616982195, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1310824, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28882, "largest_seqno": 30255, "table_properties": {"data_size": 1305495, "index_size": 2970, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 12779, "raw_average_key_size": 24, "raw_value_size": 1294512, "raw_average_value_size": 2461, "num_data_blocks": 133, "num_entries": 526, "num_filter_entries": 526, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707932616, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8d1e4347-68c9-4caa-8dac-e50739e818cb", "db_session_id": "3Q4W5E4UFNMJ922JAM41", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-02-14T17:43:37.253 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932616982285, "job": 1, "event": "recovery_finished"} 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559177200000 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: DB pointer 0x5591771ea000 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** DB Stats ** 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:43:37.254 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** Compaction Stats [default] ** 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: L0 1/0 1.25 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 467.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: L6 1/0 8.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: Sum 2/0 10.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 467.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 467.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** Compaction Stats [default] ** 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 467.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-02-14T17:43:37.255 INFO:journalctl@ceph.mon.b.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: Cumulative compaction: 0.00 GB write, 147.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: Interval compaction: 0.00 GB write, 147.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: Block cache BinnedLRUCache@0x55917710f090#2 capacity: 512.00 MB usage: 4.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: starting mon.b rank 2 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:43:37.256 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???) e3 preinit fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???).mds e1 new map 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???).mds e1 print_map 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout: e1 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout: legacy client fscid: -1 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout: 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout: No filesystems configured 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???).osd e110 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:36 smithi111 ceph-mon[68099]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:37 smithi111 bash[68079]: bb055293bff4f5e3fce301838f5eb64ce0f1016bc42d6ebbb61328fbe4065665 2024-02-14T17:43:37.257 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:37 smithi111 systemd[1]: Started Ceph mon.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:43:37.258 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:37 smithi111 ceph-mon[68099]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:37.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:37 smithi017 ceph-mon[24769]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:37.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:37 smithi017 ceph-mon[29506]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:38.160 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: mon.b calling monitor election 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: mon.a calling monitor election 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: fsmap 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: mgrmap e40: y(active, since 2m), standbys: x 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:43:38.161 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:43:38.162 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:38.162 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:38.162 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:38 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: mon.b calling monitor election 2024-02-14T17:43:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: mon.a calling monitor election 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: fsmap 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: mgrmap e40: y(active, since 2m), standbys: x 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:38.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: mon.b calling monitor election 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: mon.a calling monitor election 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: fsmap 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: mgrmap e40: y(active, since 2m), standbys: x 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:38.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:38 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:40.397 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:40 smithi111 ceph-mon[68099]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:43:40.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:40 smithi017 ceph-mon[24769]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:43:40.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:40 smithi017 ceph-mon[29506]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:43:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:43:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:43:41] "GET /metrics HTTP/1.1" 200 37989 "" "Prometheus/2.43.0" 2024-02-14T17:43:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:43:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:43:41.990+0000 7f119d0b8700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-14T17:43:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:42 smithi017 ceph-mon[24769]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:42 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:42 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:42 smithi017 ceph-mon[29506]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:42 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:42 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:42 smithi111 ceph-mon[68099]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:42 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:42 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:43 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:43 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:43.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:43 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:43.763 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:43 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:43.763 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:43 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:43.763 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:43 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:43.764 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:43 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:43.764 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:43 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:43.764 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:43 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:44 smithi017 ceph-mon[24769]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:44.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:44 smithi017 ceph-mon[29506]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:44 smithi111 ceph-mon[68099]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:46.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:46.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:46.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:43:46.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.740 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:46 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.944 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:43:46.945 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.946 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:43:46.947 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:43:46.948 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.949 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:43:46.950 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:43:46.951 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:46 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Detected new or changed devices on smithi111 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all crash 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mds 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nfs 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:43:47.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all node-exporter 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all prometheus 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all alertmanager 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all grafana 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all loki 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all promtail 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Finalizing container_image settings 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: Upgrade: Complete! 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:43:47.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:47 smithi111 ceph-mon[68099]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:43:47.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Detected new or changed devices on smithi111 2024-02-14T17:43:47.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-02-14T17:43:47.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-02-14T17:43:47.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all crash 2024-02-14T17:43:47.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all mds 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all nfs 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all node-exporter 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all prometheus 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all alertmanager 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all grafana 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all loki 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all promtail 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Finalizing container_image settings 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: Upgrade: Complete! 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-02-14T17:43:47.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[24769]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi111 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 592 B/s rd, 0 op/s 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all crash 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all mds 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all nfs 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:43:47.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all node-exporter 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all prometheus 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all alertmanager 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all grafana 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all loki 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all promtail 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Finalizing container_image settings 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: Upgrade: Complete! 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:43:47.943 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:47 smithi017 ceph-mon[29506]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:43:49.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:49 smithi017 ceph-mon[24769]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-02-14T17:43:49.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:49 smithi017 ceph-mon[29506]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-02-14T17:43:49.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:49 smithi111 ceph-mon[68099]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-02-14T17:43:51.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:51 smithi017 ceph-mon[24769]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:51.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:51 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:51.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:51 smithi017 ceph-mon[29506]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:51.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:51 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:51 smithi111 ceph-mon[68099]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:51 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:43:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:43:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:43:51] "GET /metrics HTTP/1.1" 200 38004 "" "Prometheus/2.43.0" 2024-02-14T17:43:53.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:53 smithi017 ceph-mon[29506]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-14T17:43:53.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:53 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:53.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:53 smithi017 ceph-mon[24769]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-14T17:43:53.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:53 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:53.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:53 smithi111 ceph-mon[68099]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-14T17:43:53.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:53 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:43:55.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:55 smithi017 ceph-mon[24769]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:55.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:55 smithi017 ceph-mon[29506]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:55.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:55 smithi111 ceph-mon[68099]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:57.784 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:57 smithi111 ceph-mon[68099]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:43:57.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:57 smithi017 ceph-mon[29506]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:43:57.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:57 smithi017 ceph-mon[24769]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:43:59.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:43:59 smithi017 ceph-mon[24769]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:43:59 smithi017 ceph-mon[29506]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:43:59.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:43:59 smithi111 ceph-mon[68099]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:01.777 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:01 smithi017 ceph-mon[24769]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:01.778 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:01 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:01.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:01 smithi017 ceph-mon[29506]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:01.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:01 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:01.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:01 smithi111 ceph-mon[68099]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:01.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:01 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:44:01] "GET /metrics HTTP/1.1" 200 38000 "" "Prometheus/2.43.0" 2024-02-14T17:44:03.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:44:03.636 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:03 smithi017 ceph-mon[24769]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:03.637 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:03 smithi017 ceph-mon[29506]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:03.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:03 smithi111 ceph-mon[68099]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:04.788 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:04 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:04.788 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:04 smithi017 ceph-mon[24769]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:04.789 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:04 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:04.789 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:04 smithi017 ceph-mon[29506]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:04.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:04 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:04.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:04 smithi111 ceph-mon[68099]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:05.236 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (113s) 107s ago 23m 15.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (97s) 23s ago 23m 85.4M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 unknown 107s ago 22m 58.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (2m) 23s ago 28m 452M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (12m) 107s ago 29m 504M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (29m) 107s ago 29m 79.1M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (28s) 23s ago 28m 18.3M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (28m) 107s ago 28m 56.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (2m) 107s ago 24m 7914k - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (2m) 23s ago 24m 16.6M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:44:05.239 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (27m) 107s ago 27m 55.0M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (27m) 107s ago 27m 54.7M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (27m) 107s ago 27m 52.6M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (26m) 107s ago 26m 51.6M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (26m) 23s ago 26m 55.3M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (25m) 23s ago 25m 55.3M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (25m) 23s ago 25m 51.5M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (25m) 23s ago 25m 56.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (2m) 23s ago 24m 58.1M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (23m) 107s ago 23m 77.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:44:05.240 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (22m) 23s ago 22m 78.7M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:44:05.854 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:05 smithi017 ceph-mon[24769]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:05.855 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:05 smithi017 ceph-mon[29506]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:05.879 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-02-14T17:44:05.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:05 smithi111 ceph-mon[68099]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:06.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:06 smithi017 ceph-mon[24769]: from='client.15657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:06.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:06 smithi017 ceph-mon[29506]: from='client.15657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:06.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:06 smithi111 ceph-mon[68099]: from='client.15657 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:07.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:07 smithi017 ceph-mon[24769]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:07 smithi017 ceph-mon[29506]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:07.942 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:44:07.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:07 smithi111 ceph-mon[68099]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:08.545 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:08 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/3480698257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:08.546 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:08 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/3480698257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:08.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:44:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:08 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/3480698257' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:09.767 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:09 smithi017 ceph-mon[24769]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:09.768 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:09 smithi017 ceph-mon[29506]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:09.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:09 smithi111 ceph-mon[68099]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": null, 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": false, 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "", 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": null, 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:44:10.488 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:44:11.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:44:11.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:11 smithi017 ceph-mon[24769]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:11.687 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:11 smithi017 ceph-mon[24769]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:11.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:11 smithi017 ceph-mon[29506]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:11.687 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:11 smithi017 ceph-mon[29506]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:11.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:44:11] "GET /metrics HTTP/1.1" 200 38000 "" "Prometheus/2.43.0" 2024-02-14T17:44:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:11 smithi111 ceph-mon[68099]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:11 smithi111 ceph-mon[68099]: from='client.15669 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:13.182 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:44:13.182 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:44:13.182 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:44:13.906 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:13 smithi017 ceph-mon[24769]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:13.906 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:13 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/310945152' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:44:13.906 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:13 smithi017 ceph-mon[29506]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:13.907 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:13 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/310945152' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:44:13.931 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-02-14T17:44:13.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:13 smithi111 ceph-mon[68099]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:13.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:13 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/310945152' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:44:14.921 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:14 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:14.921 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:14 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:14.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:14 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:15.565 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:15 smithi017 ceph-mon[24769]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:15.565 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:15 smithi017 ceph-mon[29506]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:15.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:15 smithi111 ceph-mon[68099]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:16 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:16 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:16.886 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:16 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:17.563 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:17 smithi017 ceph-mon[24769]: from='client.25594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:17.563 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:17 smithi017 ceph-mon[24769]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:17.563 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:17 smithi017 ceph-mon[24769]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mon", "hosts": "smithi017", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:17.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:17 smithi017 ceph-mon[29506]: from='client.25594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:17.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:17 smithi017 ceph-mon[29506]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:17.564 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:17 smithi017 ceph-mon[29506]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mon", "hosts": "smithi017", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:17 smithi111 ceph-mon[68099]: from='client.25594 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:17 smithi111 ceph-mon[68099]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:44:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:17 smithi111 ceph-mon[68099]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "mon", "hosts": "smithi017", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:18.369 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:44:18.965 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-14T17:44:19.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:19.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:44:19.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:19.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:19.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:19.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:44:19.555 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[29506]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-02-14T17:44:19.556 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:19.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:19.557 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:19 smithi017 ceph-mon[24769]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:44:19.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:19.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:44:19.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:19.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:19.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:19.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:44:19.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-02-14T17:44:19.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:19.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:19.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:19 smithi111 ceph-mon[68099]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:44:21.383 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:44:21.677 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:21 smithi017 ceph-mon[24769]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-14T17:44:21.677 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:21 smithi017 ceph-mon[29506]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-14T17:44:21.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:21 smithi111 ceph-mon[68099]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-14T17:44:21.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:44:21] "GET /metrics HTTP/1.1" 200 38006 "" "Prometheus/2.43.0" 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 2m ago 24m 15.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (114s) 40s ago 23m 85.4M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 unknown 2m ago 23m 58.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (3m) 40s ago 28m 452M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (13m) 2m ago 29m 504M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (29m) 2m ago 30m 79.1M 2048M 17.2.0 e1d6a67b021e c7c59c3a3107 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (45s) 40s ago 28m 18.3M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (28m) 2m ago 28m 56.7M 2048M 17.2.0 e1d6a67b021e 2ee2c64b201d 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (2m) 2m ago 24m 7914k - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (2m) 40s ago 24m 16.6M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:44:22.054 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (28m) 2m ago 28m 55.0M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (27m) 2m ago 27m 54.7M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (27m) 2m ago 27m 52.6M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (26m) 2m ago 26m 51.6M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (26m) 40s ago 26m 55.3M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (26m) 40s ago 26m 55.3M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (25m) 40s ago 25m 51.5M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (25m) 40s ago 25m 56.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (2m) 40s ago 24m 58.1M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (23m) 2m ago 23m 77.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:44:22.055 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (23m) 40s ago 23m 78.7M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:44:22.462 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 1 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "mds": {}, 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:44:22.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:44:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-02-14T17:44:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:44:22.464 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:44:22.464 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:44:22.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[24769]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[29506]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:22.604 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:44:22.605 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:22 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-14T17:44:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:22 smithi111 ceph-mon[68099]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:22 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:22 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:22 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:22 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:22 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:44:22.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:22 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi017", 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "0/2 daemons upgraded", 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mon daemons", 2024-02-14T17:44:22.813 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:44:22.814 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: Upgrade: Setting container_image for all mgr 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: Upgrade: It appears safe to stop mon.a 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: from='client.25618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:23.607 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[24769]: from='client.? 172.21.15.17:0/2437677363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: Upgrade: Setting container_image for all mgr 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: Upgrade: It appears safe to stop mon.a 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: from='client.25618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:23.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:23 smithi017 ceph-mon[29506]: from='client.? 172.21.15.17:0/2437677363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:44:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:44:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mgr 2024-02-14T17:44:23.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: Upgrade: It appears safe to stop mon.a 2024-02-14T17:44:23.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: from='client.25618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:23.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:23.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:23 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2437677363' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[29506]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:44:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:44:24.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:24.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[24769]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:24.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[24769]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:24.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:24.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:44:24.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:44:24.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:24 smithi017 ceph-mon[24769]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:24 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:24 smithi111 ceph-mon[68099]: from='client.25630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:24 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' 2024-02-14T17:44:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:24 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:44:24.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:24 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:44:24.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:24 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:25.614 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-mon[29506]: Upgrade: Updating mon.a 2024-02-14T17:44:25.614 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-mon[29506]: Deploying daemon mon.a on smithi017 2024-02-14T17:44:25.614 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-mon[29506]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:25.614 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-mon[24769]: Upgrade: Updating mon.a 2024-02-14T17:44:25.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-mon[24769]: Deploying daemon mon.a on smithi017 2024-02-14T17:44:25.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-mon[24769]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:25.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:25 smithi017 systemd[1]: Stopping Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:44:25.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a[24765]: 2024-02-14T17:44:25.602+0000 7f1f2fba2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:44:25.615 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a[24765]: 2024-02-14T17:44:25.602+0000 7f1f2fba2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-14T17:44:25.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:25 smithi111 ceph-mon[68099]: Upgrade: Updating mon.a 2024-02-14T17:44:25.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:25 smithi111 ceph-mon[68099]: Deploying daemon mon.a on smithi017 2024-02-14T17:44:25.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:25 smithi111 ceph-mon[68099]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:25.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:25 smithi017 podman[83408]: 2024-02-14 17:44:25.859269142 +0000 UTC m=+0.354695248 container died c7c59c3a31076f72a3622b516d5787e5f87d59813720a0039257beaf0d1ba3b3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, architecture=x86_64, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, maintainer=Guillaume Abrioux , release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, version=8) 2024-02-14T17:44:25.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:25] ENGINE Bus STOPPING 2024-02-14T17:44:25.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:44:25.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:25] ENGINE Bus STOPPED 2024-02-14T17:44:25.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:25] ENGINE Bus STARTING 2024-02-14T17:44:25.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:25] ENGINE Serving on http://:::9283 2024-02-14T17:44:25.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:25 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:25] ENGINE Bus STARTED 2024-02-14T17:44:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:26 smithi017 podman[83408]: 2024-02-14 17:44:26.220615883 +0000 UTC m=+0.716041992 container cleanup c7c59c3a31076f72a3622b516d5787e5f87d59813720a0039257beaf0d1ba3b3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, name=centos-stream, maintainer=Guillaume Abrioux , vcs-type=git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-14T17:44:26.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:26 smithi017 bash[83408]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a 2024-02-14T17:44:26.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:26 smithi017 podman[83433]: 2024-02-14 17:44:26.496905209 +0000 UTC m=+0.638844840 container remove c7c59c3a31076f72a3622b516d5787e5f87d59813720a0039257beaf0d1ba3b3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:44:26.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:26 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-14T17:44:26.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:26 smithi017 systemd[1]: Stopped Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:44:26.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:26 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service: Consumed 25.867s CPU time. 2024-02-14T17:44:27.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:26 smithi017 systemd[1]: Starting Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:44:27.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 podman[83565]: 2024-02-14 17:44:26.963863552 +0000 UTC m=+0.018113238 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:44:27.171 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 podman[83565]: 2024-02-14 17:44:27.072826882 +0000 UTC m=+0.127076558 container create c0b8c4fc87ac469fc6a314c1d295494a99631510bf1d1a46eda31bc750025d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:44:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 podman[83565]: 2024-02-14 17:44:27.275581797 +0000 UTC m=+0.329831473 container init c0b8c4fc87ac469fc6a314c1d295494a99631510bf1d1a46eda31bc750025d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-14T17:44:27.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 podman[83565]: 2024-02-14 17:44:27.278672082 +0000 UTC m=+0.332921759 container start c0b8c4fc87ac469fc6a314c1d295494a99631510bf1d1a46eda31bc750025d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:44:27.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: set uid:gid to 167:167 (ceph:ceph) 2024-02-14T17:44:27.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev), process ceph-mon, pid 2 2024-02-14T17:44:27.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: pidfile_write: ignore empty --pid-file 2024-02-14T17:44:27.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: load: jerasure load: lrc 2024-02-14T17:44:27.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: RocksDB version: 7.9.2 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Git sha 0 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Compile date 2024-02-14 11:59:27 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: DB SUMMARY 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: DB Session ID: 67ON8FIOQPPJNXL4MW0I 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: MANIFEST file: MANIFEST-000015 size: 2889 Bytes 2024-02-14T17:44:27.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000060.sst 2024-02-14T17:44:27.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000058.log size: 2318259 ; 2024-02-14T17:44:27.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:44:27.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:44:27.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:44:27.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-14T17:44:27.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:44:27.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-14T17:44:27.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.env: 0x557e8a894ae0 2024-02-14T17:44:27.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.fs: PosixFileSystem 2024-02-14T17:44:27.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.info_log: 0x557e8b9e18c0 2024-02-14T17:44:27.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:44:27.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.statistics: (nil) 2024-02-14T17:44:27.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.use_fsync: 0 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:44:27.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.db_log_dir: 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.wal_dir: 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:44:27.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.write_buffer_manager: 0x557e8b9b4dc0 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:44:27.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.unordered_write: 0 2024-02-14T17:44:27.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:44:27.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:44:27.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:44:27.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:44:27.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.row_cache: None 2024-02-14T17:44:27.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.wal_filter: None 2024-02-14T17:44:27.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:44:27.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:44:27.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:44:27.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.wal_compression: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-14T17:44:27.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_open_files: -1 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:44:27.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Compression algorithms supported: 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kZSTD supported: 0 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:44:27.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.merge_operator: 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_filter: None 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557e8b9e19c0) 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:44:27.453 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_type: 0 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_index_type: 0 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_shortening: 1 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: checksum: 4 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: no_block_cache: 0 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache: 0x557e8b9e5090 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_options: 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: capacity : 536870912 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_shard_bits : 4 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: strict_capacity_limit : 0 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_cache_compressed: (nil) 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: persistent_cache: (nil) 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size: 4096 2024-02-14T17:44:27.454 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_size_deviation: 10 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_restart_interval: 16 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: index_block_restart_interval: 1 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: metadata_block_size: 4096 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: partition_filters: 0 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: use_delta_encoding: 1 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: filter_policy: bloomfilter 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: whole_key_filtering: 1 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: verify_compression: 0 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: format_version: 5 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_index_compression: 1 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: block_align: 0 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: max_auto_readahead_size: 262144 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: prepopulate_block_cache: 0 2024-02-14T17:44:27.455 INFO:journalctl@ceph.mon.a.smithi017.stdout: initial_auto_readahead_size: 8192 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression: NoCompression 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.num_levels: 7 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:44:27.456 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:44:27.457 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:44:27.458 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.arena_block_size: 1048576 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:44:27.459 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:44:27.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.ttl: 2592000 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:44:27.461 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.blob_file_starting_level: 0 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 62, last_sequence is 26779, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4c36e236-ef52-4870-b795-549bfe26281a 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932667307563, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-02-14T17:44:27.462 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932667315395, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 2158016, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 26780, "largest_seqno": 28988, "table_properties": {"data_size": 2149559, "index_size": 5069, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2373, "raw_key_size": 23361, "raw_average_key_size": 24, "raw_value_size": 2130186, "raw_average_value_size": 2249, "num_data_blocks": 228, "num_entries": 947, "num_filter_entries": 947, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707932667, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4c36e236-ef52-4870-b795-549bfe26281a", "db_session_id": "67ON8FIOQPPJNXL4MW0I", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932667315507, "job": 1, "event": "recovery_finished"} 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557e8bad6000 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: DB pointer 0x557e8bac0000 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** DB Stats ** 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:44:27.463 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: L0 1/0 2.06 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 527.4 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: L6 1/0 8.78 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: Sum 2/0 10.84 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 527.4 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 527.4 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:44:27.464 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 527.4 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Cumulative compaction: 0.00 GB write, 200.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Interval compaction: 0.00 GB write, 200.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:44:27.465 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache BinnedLRUCache@0x557e8b9e5090#2 capacity: 512.00 MB usage: 7.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.53 KB,0.000482798%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: starting mon.a rank 0 at public addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] at bind addrs [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???) e3 preinit fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???).mds e1 new map 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???).mds e1 print_map 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: e1 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: legacy client fscid: -1 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout: No filesystems configured 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???).osd e110 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:44:27.466 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:44:27.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:44:27.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 ceph-mon[83585]: mon.a@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-02-14T17:44:27.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 bash[83565]: c0b8c4fc87ac469fc6a314c1d295494a99631510bf1d1a46eda31bc750025d7e 2024-02-14T17:44:27.467 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:27 smithi017 systemd[1]: Started Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:44:29.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ignoring --setuser ceph since I am not root 2024-02-14T17:44:29.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ignoring --setgroup ceph since I am not root 2024-02-14T17:44:29.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mgr[62961]: -- 172.21.15.17:0/1520108955 <== mon.1 v2:172.21.15.17:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cbe3dc61a0 con 0x55cbe3d9d000 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: mon.a calling monitor election 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: fsmap 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: mgrmap e40: y(active, since 3m), standbys: x 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:44:29.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: from='mgr.15306 ' entity='' 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[83585]: mgrmap e41: y(active, since 3m), standbys: x 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-14T17:44:29.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: mon.a calling monitor election 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: fsmap 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: mgrmap e40: y(active, since 3m), standbys: x 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: from='mgr.15306 ' entity='' 2024-02-14T17:44:29.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:28 smithi017 ceph-mon[29506]: mgrmap e41: y(active, since 3m), standbys: x 2024-02-14T17:44:29.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: ignoring --setuser ceph since I am not root 2024-02-14T17:44:29.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: ignoring --setgroup ceph since I am not root 2024-02-14T17:44:29.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mgr[61283]: -- 172.21.15.111:0/3929516319 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5600dc5481a0 con 0x5600dc51e800 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: from='mgr.15306 172.21.15.17:0/3673664656' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 0 op/s 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: mon.a calling monitor election 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: monmap e3: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: fsmap 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: osdmap e110: 8 total, 8 up, 8 in 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: mgrmap e40: y(active, since 3m), standbys: x 2024-02-14T17:44:29.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:44:29.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:44:29.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in unknown state 2024-02-14T17:44:29.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: from='mgr.15306 ' entity='' 2024-02-14T17:44:29.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:28 smithi111 ceph-mon[68099]: mgrmap e41: y(active, since 3m), standbys: x 2024-02-14T17:44:29.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:29 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:29.360+0000 7f1ceb468200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:44:29.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:29 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:29.458+0000 7f1ceb468200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:44:29.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:29.332+0000 7fef3df8a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-14T17:44:29.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:29.428+0000 7fef3df8a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-14T17:44:30.404 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.072+0000 7fef3df8a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:44:30.404 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.165+0000 7fef3df8a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:44:30.404 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.307+0000 7fef3df8a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:44:30.405 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.403+0000 7fef3df8a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:44:30.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:30.121+0000 7f1ceb468200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-14T17:44:30.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:30.217+0000 7f1ceb468200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-14T17:44:30.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:30.362+0000 7f1ceb468200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-14T17:44:30.734 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.486+0000 7fef3df8a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:44:30.766 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:30.460+0000 7f1ceb468200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-14T17:44:30.766 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:30.546+0000 7f1ceb468200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-14T17:44:31.021 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-mon[83585]: mgrmap e42: y(active, since 3m), standbys: x 2024-02-14T17:44:31.021 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-mon[29506]: mgrmap e42: y(active, since 3m), standbys: x 2024-02-14T17:44:31.021 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:30.817+0000 7f1ceb468200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:44:31.022 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:30 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:30.930+0000 7f1ceb468200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:44:31.022 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:31.019+0000 7f1ceb468200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:44:31.043 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-mon[68099]: mgrmap e42: y(active, since 3m), standbys: x 2024-02-14T17:44:31.043 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.750+0000 7fef3df8a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-14T17:44:31.043 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.861+0000 7fef3df8a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-14T17:44:31.044 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:30.947+0000 7fef3df8a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-14T17:44:31.044 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:31.042+0000 7fef3df8a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:44:31.299 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:31.117+0000 7f1ceb468200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-14T17:44:31.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:31.219+0000 7fef3df8a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:44:31.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:31.297+0000 7f1ceb468200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-14T17:44:32.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:32.191+0000 7fef3df8a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:44:32.484 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:32.320+0000 7fef3df8a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:44:32.591 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:32.299+0000 7f1ceb468200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-14T17:44:32.592 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:32.438+0000 7f1ceb468200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-14T17:44:32.890 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:32.788+0000 7f1ceb468200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:44:32.955 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:32.664+0000 7fef3df8a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-14T17:44:32.955 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:32.755+0000 7fef3df8a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:44:33.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:32.887+0000 7f1ceb468200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-14T17:44:33.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:33.105+0000 7f1ceb468200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:44:33.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:32.954+0000 7fef3df8a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:44:33.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:33.144+0000 7fef3df8a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:44:33.608 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:33.304+0000 7f1ceb468200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-14T17:44:33.608 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:33.606+0000 7f1ceb468200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:44:33.697 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:33.434+0000 7fef3df8a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-14T17:44:33.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:33.695+0000 7fef3df8a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:44:33.984 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:33.935+0000 7fef3df8a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:44:34.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:33.874+0000 7f1ceb468200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-14T17:44:34.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:34.123+0000 7f1ceb468200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-14T17:44:34.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:44:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:44:34.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:44:34.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:44:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:44:34.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:44:35.234 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:34.744+0000 7fef3df8a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:44:35.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:34.967+0000 7f1ceb468200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-14T17:44:35.649 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:35.320+0000 7fef3df8a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:44:35.649 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:35.424+0000 7fef3df8a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:44:35.902 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[29506]: Standby manager daemon x restarted 2024-02-14T17:44:35.902 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[29506]: Standby manager daemon x started 2024-02-14T17:44:35.902 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:44:35.902 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:44:35.902 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:44:35.902 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[29506]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[83585]: Standby manager daemon x restarted 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[83585]: Standby manager daemon x started 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[83585]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[83585]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[83585]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-mon[83585]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:35.562+0000 7f1ceb468200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:35.667+0000 7f1ceb468200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-14T17:44:35.903 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:35.900+0000 7f1ceb468200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:44:35.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-mon[68099]: Standby manager daemon x restarted 2024-02-14T17:44:35.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-mon[68099]: Standby manager daemon x started 2024-02-14T17:44:35.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-mon[68099]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-14T17:44:35.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-mon[68099]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-14T17:44:35.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-mon[68099]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-14T17:44:35.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-mon[68099]: from='mgr.? 172.21.15.111:0/427243621' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-14T17:44:35.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:35.647+0000 7fef3df8a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-14T17:44:35.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: [14/Feb/2024:17:44:35] ENGINE Bus STARTING 2024-02-14T17:44:35.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: CherryPy Checker: 2024-02-14T17:44:35.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: The Application mounted at '' has an empty config. 2024-02-14T17:44:35.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: 2024-02-14T17:44:35.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: [14/Feb/2024:17:44:35] ENGINE Serving on http://:::9283 2024-02-14T17:44:35.985 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:44:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x[61279]: [14/Feb/2024:17:44:35] ENGINE Bus STARTED 2024-02-14T17:44:36.155 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:36] ENGINE Bus STARTING 2024-02-14T17:44:36.155 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:35 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c[29502]: 2024-02-14T17:44:35.916+0000 7efd208f8700 -1 mon.c@1(peon).osd e111 update_from_paxos full map CRC mismatch, resetting to canonical 2024-02-14T17:44:36.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: CherryPy Checker: 2024-02-14T17:44:36.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: The Application mounted at '' has an empty config. 2024-02-14T17:44:36.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:36.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:36] ENGINE Serving on http://:::9283 2024-02-14T17:44:36.440 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:36] ENGINE Bus STARTED 2024-02-14T17:44:36.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: mgrmap e43: y(active, since 3m), standbys: x 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: Active manager daemon y restarted 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: Activating manager daemon y 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: failed to encode map e111 with expected crc 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: osdmap e111: 8 total, 8 up, 8 in 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: mgrmap e44: y(active, starting, since 0.00685538s), standbys: x 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: failed to encode map e111 with expected crc 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:44:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: Manager daemon y is now available 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:36.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:44:36.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:44:36.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: failed to encode map e111 with expected crc 2024-02-14T17:44:36.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:44:36.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:44:36.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: failed to encode map e111 with expected crc 2024-02-14T17:44:36.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: failed to encode map e111 with expected crc 2024-02-14T17:44:36.987 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:36 smithi111 ceph-mon[68099]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.045 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:36.926+0000 7f1ca1663700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: mgrmap e43: y(active, since 3m), standbys: x 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: Active manager daemon y restarted 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: Activating manager daemon y 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: osdmap e111: 8 total, 8 up, 8 in 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: mgrmap e44: y(active, starting, since 0.00685538s), standbys: x 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.046 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:44:37.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: Manager daemon y is now available 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.048 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[83585]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: mgrmap e43: y(active, since 3m), standbys: x 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: Active manager daemon y restarted 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: Activating manager daemon y 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: osdmap e111: 8 total, 8 up, 8 in 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: mgrmap e44: y(active, starting, since 0.00685538s), standbys: x 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:37.049 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-14T17:44:37.050 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: Manager daemon y is now available 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.051 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:36 smithi017 ceph-mon[29506]: failed to encode map e111 with expected crc 2024-02-14T17:44:37.334 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:44:37.044Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=4 2024-02-14T17:44:37.335 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:44:37.244Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=4 2024-02-14T17:44:37.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:37 smithi111 ceph-mon[68099]: [14/Feb/2024:17:44:36] ENGINE Bus STARTING 2024-02-14T17:44:37.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:37 smithi111 ceph-mon[68099]: [14/Feb/2024:17:44:36] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:44:37.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:37 smithi111 ceph-mon[68099]: [14/Feb/2024:17:44:36] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:44:37.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:37 smithi111 ceph-mon[68099]: [14/Feb/2024:17:44:36] ENGINE Bus STARTED 2024-02-14T17:44:37.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:37 smithi111 ceph-mon[68099]: mgrmap e45: y(active, since 1.00976s), standbys: x 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[29506]: [14/Feb/2024:17:44:36] ENGINE Bus STARTING 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[29506]: [14/Feb/2024:17:44:36] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[29506]: [14/Feb/2024:17:44:36] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[29506]: [14/Feb/2024:17:44:36] ENGINE Bus STARTED 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[29506]: mgrmap e45: y(active, since 1.00976s), standbys: x 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[83585]: [14/Feb/2024:17:44:36] ENGINE Bus STARTING 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[83585]: [14/Feb/2024:17:44:36] ENGINE Serving on http://172.21.15.17:8765 2024-02-14T17:44:38.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[83585]: [14/Feb/2024:17:44:36] ENGINE Serving on https://172.21.15.17:7150 2024-02-14T17:44:38.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[83585]: [14/Feb/2024:17:44:36] ENGINE Bus STARTED 2024-02-14T17:44:38.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:37 smithi017 ceph-mon[83585]: mgrmap e45: y(active, since 1.00976s), standbys: x 2024-02-14T17:44:38.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:38.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:38 smithi111 ceph-mon[68099]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:38.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:38 smithi111 ceph-mon[68099]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-14T17:44:38.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:38 smithi111 ceph-mon[68099]: Cluster is now healthy 2024-02-14T17:44:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[29506]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[29506]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-14T17:44:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[29506]: Cluster is now healthy 2024-02-14T17:44:39.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:39.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[83585]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[83585]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-02-14T17:44:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:38 smithi017 ceph-mon[83585]: Cluster is now healthy 2024-02-14T17:44:39.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:39 smithi111 ceph-mon[68099]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:39.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:39 smithi111 ceph-mon[68099]: mgrmap e46: y(active, since 3s), standbys: x 2024-02-14T17:44:40.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:39 smithi017 ceph-mon[83585]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:40.108 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:39 smithi017 ceph-mon[83585]: mgrmap e46: y(active, since 3s), standbys: x 2024-02-14T17:44:40.109 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:39 smithi017 ceph-mon[29506]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:40.109 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:39 smithi017 ceph-mon[29506]: mgrmap e46: y(active, since 3s), standbys: x 2024-02-14T17:44:41.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[29506]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:41.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.778 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.779 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[29506]: mgrmap e47: y(active, since 4s), standbys: x 2024-02-14T17:44:41.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[83585]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:41.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.779 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-mon[83585]: mgrmap e47: y(active, since 4s), standbys: x 2024-02-14T17:44:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:41 smithi111 ceph-mon[68099]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:44:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:41 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:41 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:41 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:41 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:41.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:41 smithi111 ceph-mon[68099]: mgrmap e47: y(active, since 4s), standbys: x 2024-02-14T17:44:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:44:41] "GET /metrics HTTP/1.1" 200 35201 "" "Prometheus/2.43.0" 2024-02-14T17:44:42.672 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.673 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.673 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.673 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.673 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.673 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.673 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.673 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:42 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:42 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:42 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:42 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:42.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:42 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:43.843 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:43 smithi017 ceph-mon[83585]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-14T17:44:43.843 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:43 smithi017 ceph-mon[29506]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-14T17:44:44.061 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:43 smithi111 ceph-mon[68099]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-14T17:44:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[29506]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:44:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:44:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[83585]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:44:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:44:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:45 smithi111 ceph-mon[68099]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:44:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:45 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:45 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:45 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:45 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:44:47.040 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: Detected new or changed devices on smithi017 2024-02-14T17:44:47.040 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:44:47.040 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:44:47.040 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:44:47.040 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:44:47.040 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: Detected new or changed devices on smithi017 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:44:47.041 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:44:47.042 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-14T17:44:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: Detected new or changed devices on smithi017 2024-02-14T17:44:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: Updating smithi017:/etc/ceph/ceph.conf 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: Updating smithi111:/etc/ceph/ceph.conf 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.conf 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: Updating smithi017:/etc/ceph/ceph.client.admin.keyring 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-02-14T17:44:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-02-14T17:44:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[83585]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:44:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[83585]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:44:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[83585]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:44:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[83585]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:44:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[83585]: Upgrade: It appears safe to stop mon.c 2024-02-14T17:44:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[83585]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-14T17:44:48.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[29506]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:44:48.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[29506]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:44:48.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[29506]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:44:48.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[29506]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:44:48.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[29506]: Upgrade: It appears safe to stop mon.c 2024-02-14T17:44:48.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:47 smithi017 ceph-mon[29506]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-14T17:44:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:47 smithi111 ceph-mon[68099]: Updating smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:44:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:47 smithi111 ceph-mon[68099]: Updating smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/config/ceph.client.admin.keyring 2024-02-14T17:44:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:47 smithi111 ceph-mon[68099]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:44:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:47 smithi111 ceph-mon[68099]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-14T17:44:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:47 smithi111 ceph-mon[68099]: Upgrade: It appears safe to stop mon.c 2024-02-14T17:44:48.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:47 smithi111 ceph-mon[68099]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-14T17:44:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[83585]: Upgrade: Updating mon.c 2024-02-14T17:44:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[83585]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:44:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:44:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[83585]: Deploying daemon mon.c on smithi017 2024-02-14T17:44:48.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[29506]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:48.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[29506]: Upgrade: Updating mon.c 2024-02-14T17:44:48.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[29506]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:48.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:44:48.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:44:48.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[29506]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:48.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:48 smithi017 ceph-mon[29506]: Deploying daemon mon.c on smithi017 2024-02-14T17:44:49.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:49.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:48 smithi111 ceph-mon[68099]: Upgrade: Updating mon.c 2024-02-14T17:44:49.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:48 smithi111 ceph-mon[68099]: from='mgr.25642 ' entity='mgr.y' 2024-02-14T17:44:49.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:48 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:44:49.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:48 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:44:49.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:48 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:44:49.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:48 smithi111 ceph-mon[68099]: Deploying daemon mon.c on smithi017 2024-02-14T17:44:49.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:49 smithi017 systemd[1]: Stopping Ceph mon.c for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:44:50.114 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c[29502]: 2024-02-14T17:44:49.778+0000 7efd26103700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:44:50.114 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c[29502]: 2024-02-14T17:44:49.778+0000 7efd26103700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-14T17:44:50.115 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:49 smithi017 podman[88604]: 2024-02-14 17:44:49.809854322 +0000 UTC m=+0.116375129 container died 2ee2c64b201d24dce24f88c1ab5d4e01496885ce6bab980e2cdbfa745b1b1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, version=8, io.buildah.version=1.19.8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, maintainer=Guillaume Abrioux ) 2024-02-14T17:44:50.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:49] ENGINE Bus STOPPING 2024-02-14T17:44:50.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:44:50.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:49] ENGINE Bus STOPPED 2024-02-14T17:44:50.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:49] ENGINE Bus STARTING 2024-02-14T17:44:50.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:49] ENGINE Serving on http://:::9283 2024-02-14T17:44:50.115 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:44:49] ENGINE Bus STARTED 2024-02-14T17:44:50.415 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 podman[88604]: 2024-02-14 17:44:50.212888323 +0000 UTC m=+0.519409139 container cleanup 2ee2c64b201d24dce24f88c1ab5d4e01496885ce6bab980e2cdbfa745b1b1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, name=centos-stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_CLEAN=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , version=8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vendor=Red Hat, Inc.) 2024-02-14T17:44:50.415 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 bash[88604]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c 2024-02-14T17:44:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 podman[88629]: 2024-02-14 17:44:50.414059004 +0000 UTC m=+0.605344637 container remove 2ee2c64b201d24dce24f88c1ab5d4e01496885ce6bab980e2cdbfa745b1b1b77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., version=8, name=centos-stream, GIT_BRANCH=HEAD, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-14T17:44:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-14T17:44:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 systemd[1]: Stopped Ceph mon.c for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:44:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.c.service: Consumed 15.788s CPU time. 2024-02-14T17:44:51.095 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 systemd[1]: Starting Ceph mon.c for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:44:51.096 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 podman[88758]: 2024-02-14 17:44:50.884325823 +0000 UTC m=+0.018524280 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:44:51.096 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:50 smithi017 podman[88758]: 2024-02-14 17:44:50.990197022 +0000 UTC m=+0.124395519 container create 6ea55a625d8bd76c6b19f250e2874fcd76f6b8ce8182e7c7711d2aee0f1a1121 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-14T17:44:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 podman[88758]: 2024-02-14 17:44:51.218129899 +0000 UTC m=+0.352328354 container init 6ea55a625d8bd76c6b19f250e2874fcd76f6b8ce8182e7c7711d2aee0f1a1121 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214) 2024-02-14T17:44:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 podman[88758]: 2024-02-14 17:44:51.221284438 +0000 UTC m=+0.355482892 container start 6ea55a625d8bd76c6b19f250e2874fcd76f6b8ce8182e7c7711d2aee0f1a1121 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: set uid:gid to 167:167 (ceph:ceph) 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev), process ceph-mon, pid 2 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: pidfile_write: ignore empty --pid-file 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: load: jerasure load: lrc 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: RocksDB version: 7.9.2 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Git sha 0 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Compile date 2024-02-14 11:59:27 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: DB SUMMARY 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: DB Session ID: 7P1A6X24PFY9WOBNQRSZ 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: CURRENT file: CURRENT 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: IDENTITY file: IDENTITY 2024-02-14T17:44:51.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3196 Bytes 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2681615 ; 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.error_if_exists: 0 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.create_if_missing: 0 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.paranoid_checks: 1 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.env: 0x556a89205ae0 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.fs: PosixFileSystem 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.info_log: 0x556a8ab698c0 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_file_opening_threads: 16 2024-02-14T17:44:51.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.statistics: (nil) 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.use_fsync: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_log_file_size: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.keep_log_file_num: 1000 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.recycle_log_file_num: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.allow_fallocate: 1 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.allow_mmap_reads: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.allow_mmap_writes: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.use_direct_reads: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.create_missing_column_families: 0 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.db_log_dir: 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.wal_dir: 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-14T17:44:51.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.advise_random_on_open: 1 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.db_write_buffer_size: 0 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.write_buffer_manager: 0x556a8ab3cdc0 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.rate_limiter: (nil) 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.wal_recovery_mode: 2 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.enable_thread_tracking: 0 2024-02-14T17:44:51.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.enable_pipelined_write: 0 2024-02-14T17:44:51.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.unordered_write: 0 2024-02-14T17:44:51.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-14T17:44:51.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-14T17:44:51.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-14T17:44:51.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-14T17:44:51.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.row_cache: None 2024-02-14T17:44:51.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.wal_filter: None 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.allow_ingest_behind: 0 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.two_write_queues: 0 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.manual_wal_flush: 0 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.wal_compression: 0 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.atomic_flush: 0 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-14T17:44:51.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.log_readahead_size: 0 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.best_efforts_recovery: 0 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.allow_data_in_errors: 0 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.db_host_id: __hostname__ 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_background_jobs: 2 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_background_compactions: -1 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_subcompactions: 1 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-14T17:44:51.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_total_wal_size: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_open_files: -1 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bytes_per_sync: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_readahead_size: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_background_flushes: -1 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Compression algorithms supported: 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kZSTD supported: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kXpressCompression supported: 0 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kLZ4HCCompression supported: 1 2024-02-14T17:44:51.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kZlibCompression supported: 1 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kSnappyCompression supported: 1 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kLZ4Compression supported: 1 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: kBZip2Compression supported: 0 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.merge_operator: 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_filter: None 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_filter_factory: None 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.sst_partitioner_factory: None 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556a8ab699c0) 2024-02-14T17:44:51.449 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks: 1 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: pin_top_level_index_and_filter: 1 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_type: 0 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_index_type: 0 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_shortening: 1 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: checksum: 4 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: no_block_cache: 0 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache: 0x556a8ab6d090 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_name: BinnedLRUCache 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_options: 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: capacity : 536870912 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: num_shard_bits : 4 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: strict_capacity_limit : 0 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: high_pri_pool_ratio: 0.000 2024-02-14T17:44:51.450 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_cache_compressed: (nil) 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: persistent_cache: (nil) 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size: 4096 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_size_deviation: 10 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_restart_interval: 16 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: index_block_restart_interval: 1 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: metadata_block_size: 4096 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: partition_filters: 0 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: use_delta_encoding: 1 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: filter_policy: bloomfilter 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: whole_key_filtering: 1 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: verify_compression: 0 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: read_amp_bytes_per_bit: 0 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: format_version: 5 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_index_compression: 1 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: block_align: 0 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: max_auto_readahead_size: 262144 2024-02-14T17:44:51.451 INFO:journalctl@ceph.mon.c.smithi017.stdout: prepopulate_block_cache: 0 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout: initial_auto_readahead_size: 8192 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout: num_file_reads_for_auto_readahead: 2 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.write_buffer_size: 33554432 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_write_buffer_number: 2 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression: NoCompression 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression: Disabled 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.prefix_extractor: nullptr 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.num_levels: 7 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-14T17:44:51.452 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.level: 32767 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.strategy: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.enabled: false 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-14T17:44:51.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.target_file_size_base: 67108864 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-14T17:44:51.454 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.arena_block_size: 1048576 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.disable_auto_compactions: 0 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.table_properties_collectors: 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.inplace_update_support: 0 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-14T17:44:51.455 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.bloom_locality: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.max_successive_merges: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.paranoid_file_checks: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.force_consistency_checks: 1 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.report_bg_io_stats: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.ttl: 2592000 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.enable_blob_files: false 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.min_blob_size: 0 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.blob_file_size: 268435456 2024-02-14T17:44:51.456 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.blob_file_starting_level: 0 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32127, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6ff770d3-8c5a-4162-a004-4f462f2c1f2b 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932691249766, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932691256477, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1704975, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32133, "largest_seqno": 32572, "table_properties": {"data_size": 1701918, "index_size": 1467, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 5179, "raw_average_key_size": 24, "raw_value_size": 1697097, "raw_average_value_size": 7930, "num_data_blocks": 66, "num_entries": 214, "num_filter_entries": 214, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707932691, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6ff770d3-8c5a-4162-a004-4f462f2c1f2b", "db_session_id": "7P1A6X24PFY9WOBNQRSZ", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707932691256575, "job": 1, "event": "recovery_finished"} 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-02-14T17:44:51.457 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556a8ac5e000 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: DB pointer 0x556a8ac48000 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** DB Stats ** 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:44:51.458 INFO:journalctl@ceph.mon.c.smithi017.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: L0 1/0 1.63 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 509.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: L6 1/0 9.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: Sum 2/0 11.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 509.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 509.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** Compaction Stats [default] ** 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 509.4 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-02-14T17:44:51.459 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: Cumulative compaction: 0.00 GB write, 179.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: Interval compaction: 0.00 GB write, 179.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: Block cache BinnedLRUCache@0x556a8ab6d090#2 capacity: 512.00 MB usage: 2.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,1.58 KB,0.000301003%) Misc(1,0.00 KB,0%) 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: starting mon.c rank 1 at public addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] at bind addrs [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???) e3 preinit fsid 612435b4-cb5c-11ee-95b9-87774f69a715 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???).mds e1 new map 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???).mds e1 print_map 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: e1 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-14T17:44:51.460 INFO:journalctl@ceph.mon.c.smithi017.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout: legacy client fscid: -1 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout: 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout: No filesystems configured 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???).osd e111 crush map has features 3314933000854323200, adjusting msgr requires 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-mon[88777]: mon.c@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 bash[88758]: 6ea55a625d8bd76c6b19f250e2874fcd76f6b8ce8182e7c7711d2aee0f1a1121 2024-02-14T17:44:51.461 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:51 smithi017 systemd[1]: Started Ceph mon.c for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:44:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:44:51] "GET /metrics HTTP/1.1" 200 38016 "" "Prometheus/2.43.0" 2024-02-14T17:44:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[83585]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-14T17:44:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:53.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[88777]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-14T17:44:53.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:53.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:53.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:53.194 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:44:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:52 smithi111 ceph-mon[68099]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-14T17:44:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:53.890 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:44:53.890 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (2m) 13s ago 24m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:44:53.890 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (2m) 13s ago 24m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:44:53.890 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 13s ago 23m 58.7M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:44:53.890 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (3m) 13s ago 28m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:44:53.890 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (13m) 13s ago 30m 474M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (26s) 13s ago 30m 33.9M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (77s) 13s ago 29m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 starting - - - 2048M 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (3m) 13s ago 24m 15.3M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (3m) 13s ago 24m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (28m) 13s ago 28m 55.1M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (28m) 13s ago 28m 55.5M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (27m) 13s ago 27m 52.9M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (27m) 13s ago 27m 52.4M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (27m) 13s ago 27m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (26m) 13s ago 26m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (26m) 13s ago 26m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (25m) 13s ago 25m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (2m) 13s ago 25m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (23m) 13s ago 23m 78.5M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:44:53.891 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (23m) 13s ago 23m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:44:54.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:54.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: mon.a calling monitor election 2024-02-14T17:44:54.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: mon.c calling monitor election 2024-02-14T17:44:54.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: mon.b calling monitor election 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: fsmap 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: osdmap e111: 8 total, 8 up, 8 in 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: mgrmap e47: y(active, since 16s), standbys: x 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:54.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[83585]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: mon.a calling monitor election 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: mon.c calling monitor election 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: mon.b calling monitor election 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: fsmap 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: osdmap e111: 8 total, 8 up, 8 in 2024-02-14T17:44:54.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: mgrmap e47: y(active, since 16s), standbys: x 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:54.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:53 smithi017 ceph-mon[88777]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-14T17:44:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: mon.a calling monitor election 2024-02-14T17:44:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: mon.c calling monitor election 2024-02-14T17:44:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: mon.b calling monitor election 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: monmap e4: 3 mons at {a=[v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0],b=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],c=[v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: fsmap 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: osdmap e111: 8 total, 8 up, 8 in 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: mgrmap e47: y(active, since 16s), standbys: x 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:54.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:53 smithi111 ceph-mon[68099]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:54.337 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:44:54.337 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 5 2024-02-14T17:44:54.338 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:44:54.339 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:44:54.692 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:44:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:44:54.692 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:44:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi017", 2024-02-14T17:44:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:44:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "1/2 daemons upgraded", 2024-02-14T17:44:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mon daemons", 2024-02-14T17:44:54.693 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:44:54.693 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:44:55.109 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[88777]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:55.109 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[88777]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-02-14T17:44:55.109 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1241935268' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:55.109 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[88777]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:55.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[83585]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:55.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[83585]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-02-14T17:44:55.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1241935268' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:55.110 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:54 smithi017 ceph-mon[83585]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:54 smithi111 ceph-mon[68099]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:54 smithi111 ceph-mon[68099]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-02-14T17:44:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:54 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1241935268' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:44:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:54 smithi111 ceph-mon[68099]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:44:56.444 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:44:56 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:44:56.263+0000 7f1ca1663700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-02-14T17:44:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:57 smithi017 ceph-mon[83585]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-02-14T17:44:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:57 smithi017 ceph-mon[88777]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-02-14T17:44:57.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:57 smithi111 ceph-mon[68099]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-02-14T17:44:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:44:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:59 smithi017 ceph-mon[83585]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:44:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:59 smithi017 ceph-mon[88777]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:59.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:59.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:44:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:59 smithi111 ceph-mon[68099]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:44:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:44:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:44:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:01 smithi111 ceph-mon[68099]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:01.760 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:01 smithi017 ceph-mon[83585]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:01.760 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:01 smithi017 ceph-mon[88777]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:45:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:45:01] "GET /metrics HTTP/1.1" 200 38016 "" "Prometheus/2.43.0" 2024-02-14T17:45:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-02-14T17:45:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 531 B/s rd, 0 op/s 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 531 B/s rd, 0 op/s 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:03.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 531 B/s rd, 0 op/s 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:04.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[83585]: Reconfiguring mon.a (monmap changed)... 2024-02-14T17:45:04.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[83585]: Reconfiguring daemon mon.a on smithi017 2024-02-14T17:45:04.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:04.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:04.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:45:04.102 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[88777]: Reconfiguring mon.a (monmap changed)... 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[88777]: Reconfiguring daemon mon.a on smithi017 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:45:04.103 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:03 smithi111 ceph-mon[68099]: Reconfiguring mon.a (monmap changed)... 2024-02-14T17:45:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:03 smithi111 ceph-mon[68099]: Reconfiguring daemon mon.a on smithi017 2024-02-14T17:45:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:45:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:45:04.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:04.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:04 smithi017 ceph-mon[88777]: Reconfiguring mgr.y (monmap changed)... 2024-02-14T17:45:04.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:04 smithi017 ceph-mon[88777]: Reconfiguring daemon mgr.y on smithi017 2024-02-14T17:45:04.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:04 smithi017 ceph-mon[88777]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:04.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:04 smithi017 ceph-mon[83585]: Reconfiguring mgr.y (monmap changed)... 2024-02-14T17:45:04.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:04 smithi017 ceph-mon[83585]: Reconfiguring daemon mgr.y on smithi017 2024-02-14T17:45:04.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:04 smithi017 ceph-mon[83585]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:04 smithi111 ceph-mon[68099]: Reconfiguring mgr.y (monmap changed)... 2024-02-14T17:45:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:04 smithi111 ceph-mon[68099]: Reconfiguring daemon mgr.y on smithi017 2024-02-14T17:45:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:04 smithi111 ceph-mon[68099]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:06.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: Reconfiguring daemon mon.c on smithi017 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: Reconfiguring daemon mon.c on smithi017 2024-02-14T17:45:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: Reconfiguring mon.c (monmap changed)... 2024-02-14T17:45:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: Reconfiguring daemon mon.c on smithi017 2024-02-14T17:45:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:07.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:07.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:07.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[83585]: Reconfiguring osd.0 (monmap changed)... 2024-02-14T17:45:07.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:45:07.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:07.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.0 on smithi017 2024-02-14T17:45:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[88777]: Reconfiguring osd.0 (monmap changed)... 2024-02-14T17:45:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:45:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:07 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.0 on smithi017 2024-02-14T17:45:07.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:07.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:07.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:07 smithi111 ceph-mon[68099]: Reconfiguring osd.0 (monmap changed)... 2024-02-14T17:45:07.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:45:07.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:07.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:07 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.0 on smithi017 2024-02-14T17:45:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:08.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:08 smithi111 ceph-mon[68099]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:08.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:08.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:08.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:45:08.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:08.998 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:08.998 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[88777]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:08.998 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:08.998 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:08.998 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:45:08.998 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:08.998 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:08.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[83585]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:08.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:08.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:08.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:45:08.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:09.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:09 smithi017 ceph-mon[83585]: Reconfiguring osd.1 (monmap changed)... 2024-02-14T17:45:09.966 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:09 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.1 on smithi017 2024-02-14T17:45:09.966 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:09 smithi017 ceph-mon[88777]: Reconfiguring osd.1 (monmap changed)... 2024-02-14T17:45:09.966 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:09 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.1 on smithi017 2024-02-14T17:45:09.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:09 smithi111 ceph-mon[68099]: Reconfiguring osd.1 (monmap changed)... 2024-02-14T17:45:09.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:09 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.1 on smithi017 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[83585]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[88777]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:10.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:10.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:45:10.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:10.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:10 smithi111 ceph-mon[68099]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:10.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:10.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:10.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:45:10.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:11.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:45:11] "GET /metrics HTTP/1.1" 200 38025 "" "Prometheus/2.43.0" 2024-02-14T17:45:11.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[83585]: Reconfiguring osd.2 (monmap changed)... 2024-02-14T17:45:11.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.2 on smithi017 2024-02-14T17:45:11.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:11.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:11.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:45:11.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:11.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[88777]: Reconfiguring osd.2 (monmap changed)... 2024-02-14T17:45:11.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.2 on smithi017 2024-02-14T17:45:11.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:11.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:11.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:45:11.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:11 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:11 smithi111 ceph-mon[68099]: Reconfiguring osd.2 (monmap changed)... 2024-02-14T17:45:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:11 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.2 on smithi017 2024-02-14T17:45:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:11 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:11 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:11 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:45:11.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:11 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:12.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:12 smithi017 ceph-mon[83585]: Reconfiguring osd.3 (monmap changed)... 2024-02-14T17:45:12.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:12 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.3 on smithi017 2024-02-14T17:45:12.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:12 smithi017 ceph-mon[83585]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:45:12.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:12 smithi017 ceph-mon[88777]: Reconfiguring osd.3 (monmap changed)... 2024-02-14T17:45:12.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:12 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.3 on smithi017 2024-02-14T17:45:12.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:12 smithi017 ceph-mon[88777]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:45:12.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:12 smithi111 ceph-mon[68099]: Reconfiguring osd.3 (monmap changed)... 2024-02-14T17:45:12.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:12 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.3 on smithi017 2024-02-14T17:45:12.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:12 smithi111 ceph-mon[68099]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:45:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[83585]: Reconfiguring rgw.foo.smithi017.vbwecr (monmap changed)... 2024-02-14T17:45:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:45:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[83585]: Reconfiguring daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:45:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[83585]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[88777]: Reconfiguring rgw.foo.smithi017.vbwecr (monmap changed)... 2024-02-14T17:45:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:45:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[88777]: Reconfiguring daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:45:14.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:14 smithi017 ceph-mon[88777]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:14 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:14 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:14 smithi111 ceph-mon[68099]: Reconfiguring rgw.foo.smithi017.vbwecr (monmap changed)... 2024-02-14T17:45:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:14 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:45:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:14 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:14 smithi111 ceph-mon[68099]: Reconfiguring daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:45:14.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:14 smithi111 ceph-mon[68099]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:45:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:16.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: Reconfiguring daemon mon.b on smithi111 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[83585]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: Reconfiguring daemon mon.b on smithi111 2024-02-14T17:45:16.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:16 smithi017 ceph-mon[88777]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: Reconfiguring mon.b (monmap changed)... 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: Reconfiguring daemon mon.b on smithi111 2024-02-14T17:45:16.442 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:16 smithi111 ceph-mon[68099]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:17 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:17 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:17 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:45:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:17 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:45:17.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:17 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:45:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:45:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-14T17:45:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-14T17:45:18.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:17 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:18.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: Reconfiguring mgr.x (monmap changed)... 2024-02-14T17:45:18.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: Reconfiguring daemon mgr.x on smithi111 2024-02-14T17:45:18.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:18.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:18.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:18.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:18.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:45:18.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:19.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: Reconfiguring mgr.x (monmap changed)... 2024-02-14T17:45:19.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: Reconfiguring daemon mgr.x on smithi111 2024-02-14T17:45:19.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:19.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:19.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:19.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:19.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:45:19.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:19.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: Reconfiguring mgr.x (monmap changed)... 2024-02-14T17:45:19.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: Reconfiguring daemon mgr.x on smithi111 2024-02-14T17:45:19.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:19.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:19.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:19.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:19.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:45:19.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:19.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:19 smithi111 ceph-mon[68099]: Reconfiguring osd.4 (monmap changed)... 2024-02-14T17:45:19.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:19 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.4 on smithi111 2024-02-14T17:45:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:19 smithi017 ceph-mon[83585]: Reconfiguring osd.4 (monmap changed)... 2024-02-14T17:45:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:19 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.4 on smithi111 2024-02-14T17:45:20.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:19 smithi017 ceph-mon[88777]: Reconfiguring osd.4 (monmap changed)... 2024-02-14T17:45:20.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:19 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.4 on smithi111 2024-02-14T17:45:20.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:20 smithi111 ceph-mon[68099]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:20.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:20.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:20.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:45:20.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[83585]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:45:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[88777]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:45:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:45:21] "GET /metrics HTTP/1.1" 200 38024 "" "Prometheus/2.43.0" 2024-02-14T17:45:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[83585]: Reconfiguring osd.5 (monmap changed)... 2024-02-14T17:45:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.5 on smithi111 2024-02-14T17:45:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[88777]: Reconfiguring osd.5 (monmap changed)... 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.5 on smithi111 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:45:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:21 smithi111 ceph-mon[68099]: Reconfiguring osd.5 (monmap changed)... 2024-02-14T17:45:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:21 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.5 on smithi111 2024-02-14T17:45:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:22.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:45:22.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:23.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:22 smithi111 ceph-mon[68099]: Reconfiguring osd.6 (monmap changed)... 2024-02-14T17:45:23.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:22 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.6 on smithi111 2024-02-14T17:45:23.137 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:22 smithi111 ceph-mon[68099]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:22 smithi017 ceph-mon[83585]: Reconfiguring osd.6 (monmap changed)... 2024-02-14T17:45:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:22 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.6 on smithi111 2024-02-14T17:45:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:22 smithi017 ceph-mon[83585]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:22 smithi017 ceph-mon[88777]: Reconfiguring osd.6 (monmap changed)... 2024-02-14T17:45:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:22 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.6 on smithi111 2024-02-14T17:45:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:22 smithi017 ceph-mon[88777]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:24 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:24 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:24 smithi111 ceph-mon[68099]: Reconfiguring osd.7 (monmap changed)... 2024-02-14T17:45:24.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:24 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:45:24.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:24 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:24.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:24 smithi111 ceph-mon[68099]: Reconfiguring daemon osd.7 on smithi111 2024-02-14T17:45:24.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:24 smithi111 ceph-mon[68099]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[88777]: Reconfiguring osd.7 (monmap changed)... 2024-02-14T17:45:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:45:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[88777]: Reconfiguring daemon osd.7 on smithi111 2024-02-14T17:45:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[88777]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:24.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:24.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:24.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[83585]: Reconfiguring osd.7 (monmap changed)... 2024-02-14T17:45:24.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:45:24.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:24.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[83585]: Reconfiguring daemon osd.7 on smithi111 2024-02-14T17:45:24.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:24 smithi017 ceph-mon[83585]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:25.060 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:45:25.727 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:45:25.727 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 28s ago 25m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:45:25.727 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (2m) 45s ago 24m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:45:25.727 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 28s ago 24m 59.1M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:45:25.727 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (4m) 45s ago 29m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:45:25.727 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (14m) 28s ago 30m 494M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (58s) 28s ago 31m 42.9M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (109s) 45s ago 29m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (34s) 28s ago 29m 19.1M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (3m) 28s ago 25m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (3m) 45s ago 25m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (29m) 28s ago 29m 54.2M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (28m) 28s ago 28m 55.6M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (28m) 28s ago 28m 52.8M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (27m) 28s ago 27m 52.4M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (27m) 45s ago 27m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:45:25.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (27m) 45s ago 27m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:45:25.729 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (26m) 45s ago 26m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:45:25.729 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (26m) 45s ago 26m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:45:25.729 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (3m) 45s ago 25m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:45:25.729 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (24m) 28s ago 24m 79.1M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:45:25.729 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (24m) 45s ago 24m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: Reconfiguring rgw.foo.smithi111.zvlcwh (monmap changed)... 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: Reconfiguring daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.118 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:25 smithi111 ceph-mon[68099]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:26.133 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:45:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:45:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 5 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:45:26.134 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: Reconfiguring rgw.foo.smithi111.zvlcwh (monmap changed)... 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: Reconfiguring daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.135 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[83585]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: Reconfiguring rgw.foo.smithi111.zvlcwh (monmap changed)... 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: Reconfiguring daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: from='client.54115 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:26.136 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:25 smithi017 ceph-mon[88777]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi017", 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/2 daemons upgraded", 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading mon daemons", 2024-02-14T17:45:26.475 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:45:26.476 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:45:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/3866050649' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mon 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all crash 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mds 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nfs 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/3866050649' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mon 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all crash 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mds 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.194 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nfs 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:45:27.195 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all node-exporter 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all prometheus 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all alertmanager 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all grafana 2024-02-14T17:45:27.196 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all loki 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all promtail 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Finalizing container_image settings 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.197 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all node-exporter 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all prometheus 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all alertmanager 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.198 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all grafana 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all loki 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all promtail 2024-02-14T17:45:27.199 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Finalizing container_image settings 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:45:27.200 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.201 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:45:27.202 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.203 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: Upgrade: Complete! 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-02-14T17:45:27.204 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: Upgrade: Complete! 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:27.205 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:27.206 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/3866050649' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mon 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all crash 2024-02-14T17:45:27.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mds 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nfs 2024-02-14T17:45:27.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:45:27.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.237 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all node-exporter 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all prometheus 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all alertmanager 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all grafana 2024-02-14T17:45:27.238 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all loki 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all promtail 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Finalizing container_image settings 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:45:27.239 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:45:27.240 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: Upgrade: Complete! 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-02-14T17:45:27.241 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:27.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:45:27.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:45:27.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:45:27.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 746 B/s rd, 0 op/s 2024-02-14T17:45:27.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:27.242 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:28.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:28.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:28.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:29.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:29 smithi017 ceph-mon[83585]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:29.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:29 smithi017 ceph-mon[88777]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:29.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:29 smithi111 ceph-mon[68099]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:32.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:31 smithi017 ceph-mon[83585]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:32.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:31 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:32.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:45:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:45:31] "GET /metrics HTTP/1.1" 200 38024 "" "Prometheus/2.43.0" 2024-02-14T17:45:32.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:31 smithi017 ceph-mon[88777]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:32.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:31 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:32.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:31 smithi111 ceph-mon[68099]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-14T17:45:32.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:31 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:45:34.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:33 smithi017 ceph-mon[83585]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-14T17:45:34.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:33 smithi017 ceph-mon[88777]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-14T17:45:34.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:33 smithi111 ceph-mon[68099]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-14T17:45:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:35 smithi017 ceph-mon[83585]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:36.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:35 smithi017 ceph-mon[88777]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:35 smithi111 ceph-mon[68099]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:37.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:38.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:37 smithi017 ceph-mon[83585]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:45:38.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:37 smithi017 ceph-mon[88777]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:45:38.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:37 smithi111 ceph-mon[68099]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:45:39.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:40.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:39 smithi017 ceph-mon[88777]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:40.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:39 smithi017 ceph-mon[83585]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:40.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:39 smithi111 ceph-mon[68099]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:42.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:41 smithi017 ceph-mon[83585]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:42.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:41 smithi017 ceph-mon[88777]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:42.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:45:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:45:41] "GET /metrics HTTP/1.1" 200 38021 "" "Prometheus/2.43.0" 2024-02-14T17:45:42.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:41 smithi111 ceph-mon[68099]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:43 smithi017 ceph-mon[83585]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:44.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:43 smithi017 ceph-mon[88777]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:44.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:43 smithi111 ceph-mon[68099]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:46.036 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:45 smithi111 ceph-mon[68099]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:45 smithi017 ceph-mon[83585]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:46.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:45 smithi017 ceph-mon[88777]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:48.179 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:47 smithi111 ceph-mon[68099]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:47 smithi017 ceph-mon[83585]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:48.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:47 smithi017 ceph-mon[88777]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:49.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:49.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:48 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:49.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:50.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:49 smithi017 ceph-mon[83585]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:49 smithi017 ceph-mon[88777]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:50.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:49 smithi111 ceph-mon[68099]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:52.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:51 smithi017 ceph-mon[88777]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:52.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:51 smithi017 ceph-mon[83585]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:52.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:52.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:45:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:45:51] "GET /metrics HTTP/1.1" 200 38020 "" "Prometheus/2.43.0" 2024-02-14T17:45:52.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:51 smithi111 ceph-mon[68099]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:45:52.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:45:54.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:53 smithi017 ceph-mon[83585]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:54.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:53 smithi017 ceph-mon[88777]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:53 smithi111 ceph-mon[68099]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:56.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:55 smithi017 ceph-mon[88777]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:56.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:55 smithi017 ceph-mon[83585]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:55 smithi111 ceph-mon[68099]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:57.540 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:45:57.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:57 smithi111 ceph-mon[68099]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:57.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:57 smithi111 ceph-mon[68099]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:58.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:57 smithi017 ceph-mon[83585]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:58.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:57 smithi017 ceph-mon[83585]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:58.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:57 smithi017 ceph-mon[88777]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:45:58.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:57 smithi017 ceph-mon[88777]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:45:59.039 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:59.039 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:59.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:45:59.590 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (3m) 62s ago 25m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (3m) 79s ago 25m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 62s ago 24m 59.1M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (4m) 79s ago 29m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (14m) 62s ago 31m 494M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (92s) 62s ago 31m 42.9M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (2m) 79s ago 30m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (68s) 62s ago 30m 19.1M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 62s ago 25m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (4m) 79s ago 25m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (29m) 62s ago 29m 54.2M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (29m) 62s ago 29m 55.6M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (28m) 62s ago 28m 52.8M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:45:59.591 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (28m) 62s ago 28m 52.4M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:45:59.592 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (28m) 79s ago 28m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:45:59.592 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (27m) 79s ago 27m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:45:59.592 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (27m) 79s ago 27m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:45:59.592 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (27m) 79s ago 27m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:45:59.592 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (4m) 79s ago 26m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:45:59.592 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (24m) 62s ago 24m 79.1M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:45:59.592 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (24m) 79s ago 24m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:45:59.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:45:59 smithi017 ceph-mon[83585]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:45:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:45:59 smithi017 ceph-mon[88777]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:45:59 smithi111 ceph-mon[68099]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:00.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-02-14T17:46:00.831 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:00 smithi017 ceph-mon[83585]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:00.832 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:00 smithi017 ceph-mon[88777]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:00 smithi111 ceph-mon[68099]: from='client.54139 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:02.047 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:01 smithi017 ceph-mon[83585]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:02.047 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:01 smithi017 ceph-mon[88777]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:02.048 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:46:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:46:01] "GET /metrics HTTP/1.1" 200 38020 "" "Prometheus/2.43.0" 2024-02-14T17:46:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:01 smithi111 ceph-mon[68099]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:02.457 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:46:03.034 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:02 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/3967003648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:03.035 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:02 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/3967003648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:03.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-02-14T17:46:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:02 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/3967003648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:03.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:03 smithi017 ceph-mon[83585]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:03.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:03 smithi017 ceph-mon[88777]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:03 smithi111 ceph-mon[68099]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:05.386 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)" 2024-02-14T17:46:05.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:05 smithi017 ceph-mon[83585]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:05.982 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:05 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/867117789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:05.982 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:05 smithi017 ceph-mon[88777]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:05.982 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:05 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/867117789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:06.008 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-02-14T17:46:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:05 smithi111 ceph-mon[68099]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:05 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/867117789' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:06.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:06.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:07.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:07 smithi017 ceph-mon[83585]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:07.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:07 smithi017 ceph-mon[88777]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:08.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:07 smithi111 ceph-mon[68099]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:09.071 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:09.071 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:09.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:09.984 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:46:10.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:09 smithi017 ceph-mon[83585]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:10.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:09 smithi017 ceph-mon[83585]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:10.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:09 smithi017 ceph-mon[88777]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:10.074 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:09 smithi017 ceph-mon[88777]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:10.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:09 smithi111 ceph-mon[68099]: from='client.44220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:10.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:09 smithi111 ceph-mon[68099]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:10.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:46:12.073 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:11 smithi017 ceph-mon[83585]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:12.073 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:11 smithi017 ceph-mon[88777]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:12.073 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:46:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:46:11] "GET /metrics HTTP/1.1" 200 38022 "" "Prometheus/2.43.0" 2024-02-14T17:46:12.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:11 smithi111 ceph-mon[68099]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:12.537 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:46:12.537 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": null, 2024-02-14T17:46:12.538 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": false, 2024-02-14T17:46:12.538 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "", 2024-02-14T17:46:12.538 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:46:12.538 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": null, 2024-02-14T17:46:12.538 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-02-14T17:46:12.538 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:46:12.538 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:46:13.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:46:14.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:13 smithi017 ceph-mon[83585]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:14.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:13 smithi017 ceph-mon[83585]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:13 smithi017 ceph-mon[88777]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:13 smithi017 ceph-mon[88777]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:13 smithi111 ceph-mon[68099]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:14.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:13 smithi111 ceph-mon[68099]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:15.166 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:46:15.166 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:46:15.166 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:46:15.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-02-14T17:46:16.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:15 smithi017 ceph-mon[83585]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:16.069 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:15 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1614323404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:46:16.069 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:15 smithi017 ceph-mon[88777]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:16.069 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:15 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1614323404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:46:16.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:15 smithi111 ceph-mon[68099]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:16.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:15 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1614323404' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:46:18.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:17 smithi017 ceph-mon[83585]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:18.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:17 smithi017 ceph-mon[88777]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:18.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:17 smithi111 ceph-mon[68099]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:19.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:19.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:18 smithi017 ceph-mon[88777]: from='client.54175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:19.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:19.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:18 smithi017 ceph-mon[83585]: from='client.54175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:19.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:18 smithi111 ceph-mon[68099]: from='client.54175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:19.615 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:46:20.164 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:19 smithi017 ceph-mon[88777]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:20.164 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:19 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:20.164 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:19 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:20.164 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:19 smithi017 ceph-mon[83585]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:20.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:19 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:20.165 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:19 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:20.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-14T17:46:20.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:19 smithi111 ceph-mon[68099]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:20.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:19 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:20.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:19 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[83585]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:46:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[88777]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:46:21.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:20 smithi111 ceph-mon[68099]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:46:21.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[83585]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:21.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:21.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:21.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:46:21.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.884 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[88777]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:21.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:21.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:21.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:46:21.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:21.885 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:46:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:46:21] "GET /metrics HTTP/1.1" 200 38019 "" "Prometheus/2.43.0" 2024-02-14T17:46:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:21 smithi111 ceph-mon[68099]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:22.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:22.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:46:22.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:22.351 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:46:22.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:22 smithi017 ceph-mon[83585]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-14T17:46:22.996 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:22 smithi017 ceph-mon[83585]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-02-14T17:46:22.997 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:22 smithi017 ceph-mon[83585]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:46:22.997 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:22 smithi017 ceph-mon[88777]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-14T17:46:22.997 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:22 smithi017 ceph-mon[88777]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-02-14T17:46:22.997 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:22 smithi017 ceph-mon[88777]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:46:23.041 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:46:23.041 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (4m) 85s ago 26m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:46:23.041 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (3m) 102s ago 25m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:46:23.041 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 85s ago 25m 59.1M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:46:23.042 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (5m) 102s ago 30m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:46:23.042 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (15m) 85s ago 31m 494M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:46:23.042 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (116s) 85s ago 32m 42.9M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:46:23.042 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (2m) 102s ago 30m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:46:23.042 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (92s) 85s ago 30m 19.1M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:46:23.042 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (4m) 85s ago 26m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:46:23.043 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (4m) 102s ago 26m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:46:23.043 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (30m) 85s ago 30m 54.2M 2993M 17.2.0 e1d6a67b021e c8899980cb2f 2024-02-14T17:46:23.043 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (29m) 85s ago 29m 55.6M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:46:23.043 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (29m) 85s ago 29m 52.8M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:46:23.043 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (28m) 85s ago 28m 52.4M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:46:23.043 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (28m) 102s ago 28m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:46:23.043 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (28m) 102s ago 28m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:46:23.044 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (27m) 102s ago 27m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:46:23.044 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (27m) 102s ago 27m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:46:23.044 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (4m) 102s ago 26m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:46:23.044 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (25m) 85s ago 25m 79.1M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:46:23.044 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (25m) 102s ago 25m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:46:23.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:22 smithi111 ceph-mon[68099]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 973 B/s rd, 0 op/s 2024-02-14T17:46:23.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:22 smithi111 ceph-mon[68099]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-02-14T17:46:23.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:22 smithi111 ceph-mon[68099]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:46:23.452 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 5 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:46:23.453 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:46:23.807 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:46:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:46:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:46:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-02-14T17:46:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:46:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "0/8 daemons upgraded", 2024-02-14T17:46:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:46:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:46:23.808 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:46:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='client.44244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/4283531123' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='client.44244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/4283531123' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:24.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:23 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-14T17:46:24.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='client.44244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:24.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/4283531123' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:24.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:23 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: from='client.44256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mgr 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mon 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all crash 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-14T17:46:25.115 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: Upgrade: osd.0 is safe to restart 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[83585]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: from='client.44256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mgr 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mon 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all crash 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: Upgrade: osd.0 is safe to restart 2024-02-14T17:46:25.116 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:24 smithi017 ceph-mon[88777]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:25.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: from='client.44256 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:25.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:25.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:46:25.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:46:25.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mgr 2024-02-14T17:46:25.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mon 2024-02-14T17:46:25.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all crash 2024-02-14T17:46:25.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-02-14T17:46:25.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: Upgrade: osd.0 is safe to restart 2024-02-14T17:46:25.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:24 smithi111 ceph-mon[68099]: from='client.44268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:26.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[83585]: Upgrade: Updating osd.0 2024-02-14T17:46:26.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:26.430 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[83585]: Deploying daemon osd.0 on smithi017 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[83585]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[88777]: Upgrade: Updating osd.0 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[88777]: Deploying daemon osd.0 on smithi017 2024-02-14T17:46:26.431 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:26 smithi017 ceph-mon[88777]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:26 smithi111 ceph-mon[68099]: Upgrade: Updating osd.0 2024-02-14T17:46:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-14T17:46:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:26.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:26 smithi111 ceph-mon[68099]: Deploying daemon osd.0 on smithi017 2024-02-14T17:46:26.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:26 smithi111 ceph-mon[68099]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:28 smithi017 ceph-mon[83585]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:28.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:28.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:28 smithi017 ceph-mon[88777]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:28.691 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:28 smithi017 systemd[1]: Stopping Ceph osd.0 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:46:28.691 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[32363]: 2024-02-14T17:46:28.482+0000 7f41e46c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:46:28.691 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[32363]: 2024-02-14T17:46:28.482+0000 7f41e46c5700 -1 osd.0 111 *** Got signal Terminated *** 2024-02-14T17:46:28.691 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:28 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[32363]: 2024-02-14T17:46:28.482+0000 7f41e46c5700 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:46:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:28.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:28 smithi111 ceph-mon[68099]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:29 smithi017 ceph-mon[83585]: osd.0 marked itself down and dead 2024-02-14T17:46:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:29 smithi017 ceph-mon[88777]: osd.0 marked itself down and dead 2024-02-14T17:46:29.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:29 smithi111 ceph-mon[68099]: osd.0 marked itself down and dead 2024-02-14T17:46:30.084 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:29 smithi017 podman[96061]: 2024-02-14 17:46:29.727560933 +0000 UTC m=+1.333124882 container died c8899980cb2f8032f8ceb81c5780ddf5d5771637d9796dcb22dfc4e450949d90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, GIT_CLEAN=True, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-02-14T17:46:30.084 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:30 smithi017 podman[96061]: 2024-02-14 17:46:30.082276812 +0000 UTC m=+1.687840752 container cleanup c8899980cb2f8032f8ceb81c5780ddf5d5771637d9796dcb22dfc4e450949d90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, distribution-scope=public, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.display-name=CentOS Stream 8) 2024-02-14T17:46:30.358 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:30 smithi017 bash[96061]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0 2024-02-14T17:46:30.690 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:30 smithi017 podman[96078]: 2024-02-14 17:46:30.384049509 +0000 UTC m=+0.656216842 container remove c8899980cb2f8032f8ceb81c5780ddf5d5771637d9796dcb22dfc4e450949d90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, GIT_BRANCH=HEAD, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , release=754, architecture=x86_64, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:46:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[83585]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:46:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[83585]: failed to encode map e112 with expected crc 2024-02-14T17:46:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[83585]: osdmap e112: 8 total, 7 up, 8 in 2024-02-14T17:46:30.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[83585]: failed to encode map e112 with expected crc 2024-02-14T17:46:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[88777]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:46:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[88777]: failed to encode map e112 with expected crc 2024-02-14T17:46:30.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[88777]: osdmap e112: 8 total, 7 up, 8 in 2024-02-14T17:46:30.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:30 smithi017 ceph-mon[88777]: failed to encode map e112 with expected crc 2024-02-14T17:46:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:30 smithi111 ceph-mon[68099]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:30 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:46:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:30 smithi111 ceph-mon[68099]: failed to encode map e112 with expected crc 2024-02-14T17:46:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:30 smithi111 ceph-mon[68099]: osdmap e112: 8 total, 7 up, 8 in 2024-02-14T17:46:30.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:30 smithi111 ceph-mon[68099]: failed to encode map e112 with expected crc 2024-02-14T17:46:31.013 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:30 smithi017 podman[96169]: 2024-02-14 17:46:30.648999687 +0000 UTC m=+0.017914420 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:46:31.013 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:30 smithi017 podman[96169]: 2024-02-14 17:46:30.760234848 +0000 UTC m=+0.129149617 container create a905678e392b179ffbb638a5a72dad03767f0407e692827c3309253e1b9bbd2a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:46:31.014 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:30 smithi017 podman[96169]: 2024-02-14 17:46:30.93831467 +0000 UTC m=+0.307229405 container init a905678e392b179ffbb638a5a72dad03767f0407e692827c3309253e1b9bbd2a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-14T17:46:31.014 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:30 smithi017 podman[96169]: 2024-02-14 17:46:30.941919051 +0000 UTC m=+0.310833783 container start a905678e392b179ffbb638a5a72dad03767f0407e692827c3309253e1b9bbd2a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214) 2024-02-14T17:46:31.014 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:31 smithi017 podman[96169]: 2024-02-14 17:46:31.012412322 +0000 UTC m=+0.381327048 container attach a905678e392b179ffbb638a5a72dad03767f0407e692827c3309253e1b9bbd2a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-02-14T17:46:31.360 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:31 smithi017 podman[96169]: 2024-02-14 17:46:31.244640539 +0000 UTC m=+0.613555257 container died a905678e392b179ffbb638a5a72dad03767f0407e692827c3309253e1b9bbd2a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-14T17:46:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e112 with expected crc 2024-02-14T17:46:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: osdmap e113: 8 total, 7 up, 8 in 2024-02-14T17:46:31.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e112 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: osdmap e113: 8 total, 7 up, 8 in 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e112 with expected crc 2024-02-14T17:46:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: osdmap e113: 8 total, 7 up, 8 in 2024-02-14T17:46:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:31.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:31 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:32.119 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:46:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:46:31] "GET /metrics HTTP/1.1" 200 38019 "" "Prometheus/2.43.0" 2024-02-14T17:46:32.119 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:31 smithi017 podman[96196]: 2024-02-14 17:46:31.900276219 +0000 UTC m=+0.649807361 container remove a905678e392b179ffbb638a5a72dad03767f0407e692827c3309253e1b9bbd2a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-14T17:46:32.119 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:31 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-14T17:46:32.119 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:31 smithi017 systemd[1]: Stopped Ceph osd.0 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:46:32.119 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:31 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.0.service: Consumed 14.116s CPU time. 2024-02-14T17:46:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:32 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:32 smithi017 ceph-mon[88777]: failed to encode map e113 with expected crc 2024-02-14T17:46:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:32 smithi017 ceph-mon[88777]: pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:46:32.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:32 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:32.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:32 smithi017 ceph-mon[83585]: failed to encode map e113 with expected crc 2024-02-14T17:46:32.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:32 smithi017 ceph-mon[83585]: pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:46:32.441 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:32 smithi017 systemd[1]: Starting Ceph osd.0 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:46:32.441 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:32 smithi017 podman[96309]: 2024-02-14 17:46:32.23918997 +0000 UTC m=+0.018777368 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:46:32.441 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:32 smithi017 podman[96309]: 2024-02-14 17:46:32.35954469 +0000 UTC m=+0.139132086 container create 3b1ad62d267b1f67858133bf8a560a9d919d8715afc90974623f8b940fab8d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-14T17:46:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:32 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:32 smithi111 ceph-mon[68099]: failed to encode map e113 with expected crc 2024-02-14T17:46:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:32 smithi111 ceph-mon[68099]: pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:46:32.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:32 smithi017 podman[96309]: 2024-02-14 17:46:32.599980161 +0000 UTC m=+0.379567561 container init 3b1ad62d267b1f67858133bf8a560a9d919d8715afc90974623f8b940fab8d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:46:32.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:32 smithi017 podman[96309]: 2024-02-14 17:46:32.60322645 +0000 UTC m=+0.382813852 container start 3b1ad62d267b1f67858133bf8a560a9d919d8715afc90974623f8b940fab8d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True) 2024-02-14T17:46:32.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:32 smithi017 podman[96309]: 2024-02-14 17:46:32.699814839 +0000 UTC m=+0.479402245 container attach 3b1ad62d267b1f67858133bf8a560a9d919d8715afc90974623f8b940fab8d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-14T17:46:33.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate[96324]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-14T17:46:33.354 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate[96324]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 bash[96309]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 bash[96309]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 bash[96309]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 bash[96309]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 bash[96309]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 bash[96309]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 bash[96309]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate[96324]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate[96324]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate[96324]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate[96324]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate[96324]: --> ceph-volume raw activate successful for osd ID: 0 2024-02-14T17:46:33.355 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 podman[96309]: 2024-02-14 17:46:33.067606976 +0000 UTC m=+0.847194380 container died 3b1ad62d267b1f67858133bf8a560a9d919d8715afc90974623f8b940fab8d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214) 2024-02-14T17:46:33.621 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 21 pgs degraded (PG_DEGRADED) 2024-02-14T17:46:33.621 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:33 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 21 pgs degraded (PG_DEGRADED) 2024-02-14T17:46:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:33 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 69/627 objects degraded (11.005%), 21 pgs degraded (PG_DEGRADED) 2024-02-14T17:46:33.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 podman[96443]: 2024-02-14 17:46:33.620153545 +0000 UTC m=+0.547894283 container remove 3b1ad62d267b1f67858133bf8a560a9d919d8715afc90974623f8b940fab8d49 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.license=GPLv2) 2024-02-14T17:46:33.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 podman[96509]: 2024-02-14 17:46:33.732983079 +0000 UTC m=+0.017770102 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:46:33.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:33 smithi017 podman[96509]: 2024-02-14 17:46:33.846144143 +0000 UTC m=+0.130931161 container create 15f522343f507c077320a56977b9c85f0120fddbf8e9842db9d34f1c0dc1e02f (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:46:34.417 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:34 smithi017 podman[96509]: 2024-02-14 17:46:34.093420158 +0000 UTC m=+0.378207177 container init 15f522343f507c077320a56977b9c85f0120fddbf8e9842db9d34f1c0dc1e02f (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-14T17:46:34.417 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:34 smithi017 podman[96509]: 2024-02-14 17:46:34.096373749 +0000 UTC m=+0.381160769 container start 15f522343f507c077320a56977b9c85f0120fddbf8e9842db9d34f1c0dc1e02f (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-14T17:46:34.417 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-osd[96528]: -- 172.21.15.17:0/3539882983 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55665e0691e0 con 0x55665e03b000 2024-02-14T17:46:34.417 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:34 smithi017 bash[96509]: 15f522343f507c077320a56977b9c85f0120fddbf8e9842db9d34f1c0dc1e02f 2024-02-14T17:46:34.417 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:34 smithi017 systemd[1]: Started Ceph osd.0 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:46:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[88777]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:34.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[83585]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:34.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:34.691 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[96524]: 2024-02-14T17:46:34.637+0000 7f27c8e0b7c0 -1 Falling back to public interface 2024-02-14T17:46:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:34 smithi111 ceph-mon[68099]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:34 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:34 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:34 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:34 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:36 smithi017 ceph-mon[83585]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:36 smithi017 ceph-mon[88777]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:36 smithi111 ceph-mon[68099]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:37.659 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:37 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[96524]: 2024-02-14T17:46:37.387+0000 7f27c8e0b7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-02-14T17:46:37.659 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:37 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[96524]: 2024-02-14T17:46:37.414+0000 7f27c8e0b7c0 -1 osd.0 111 log_to_monitors true 2024-02-14T17:46:38.569 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:38.569 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:38 smithi017 ceph-mon[83585]: pgmap v73: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:38.569 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:38 smithi017 ceph-mon[83585]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:46:38.570 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:38.570 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:38 smithi017 ceph-mon[88777]: pgmap v73: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:38.570 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:38 smithi017 ceph-mon[88777]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:46:38.570 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:46:38 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[96524]: 2024-02-14T17:46:38.381+0000 7f27ba263700 -1 osd.0 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:46:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:38 smithi111 ceph-mon[68099]: pgmap v73: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:38 smithi111 ceph-mon[68099]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: osdmap e114: 8 total, 7 up, 8 in 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:39.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[88777]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: osdmap e114: 8 total, 7 up, 8 in 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:39.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:39 smithi017 ceph-mon[83585]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-14T17:46:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: osdmap e114: 8 total, 7 up, 8 in 2024-02-14T17:46:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: from='osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:46:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: failed to encode map e114 with expected crc 2024-02-14T17:46:39.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:39.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:39.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:39 smithi111 ceph-mon[68099]: failed to encode map e114 with expected crc 2024-02-14T17:46:40.654 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: failed to encode map e114 with expected crc 2024-02-14T17:46:40.654 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:40.654 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: failed to encode map e115 with expected crc 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: failed to encode map e115 with expected crc 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982] boot 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: osdmap e115: 8 total, 8 up, 8 in 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: failed to encode map e114 with expected crc 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: failed to encode map e115 with expected crc 2024-02-14T17:46:40.655 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: failed to encode map e115 with expected crc 2024-02-14T17:46:40.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982] boot 2024-02-14T17:46:40.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: osdmap e115: 8 total, 8 up, 8 in 2024-02-14T17:46:40.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:46:40.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:40.656 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: failed to encode map e114 with expected crc 2024-02-14T17:46:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: pgmap v75: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:46:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:46:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: failed to encode map e115 with expected crc 2024-02-14T17:46:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: failed to encode map e115 with expected crc 2024-02-14T17:46:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: osd.0 [v2:172.21.15.17:6802/3940989982,v1:172.21.15.17:6803/3940989982] boot 2024-02-14T17:46:40.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: osdmap e115: 8 total, 8 up, 8 in 2024-02-14T17:46:40.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-14T17:46:40.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:40.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:41.663 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.663 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.663 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: osdmap e116: 8 total, 8 up, 8 in 2024-02-14T17:46:41.663 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.663 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.663 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[88777]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: osdmap e116: 8 total, 8 up, 8 in 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.664 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.665 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-mon[83585]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: osdmap e116: 8 total, 8 up, 8 in 2024-02-14T17:46:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:41 smithi111 ceph-mon[68099]: failed to encode map e116 with expected crc 2024-02-14T17:46:41.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:46:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:46:41] "GET /metrics HTTP/1.1" 200 38146 "" "Prometheus/2.43.0" 2024-02-14T17:46:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:42 smithi017 ceph-mon[83585]: pgmap v78: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-14T17:46:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:42 smithi017 ceph-mon[88777]: pgmap v78: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-14T17:46:42.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:42 smithi111 ceph-mon[68099]: pgmap v78: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/627 objects degraded (11.005%) 2024-02-14T17:46:43.690 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:46:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:46:43.497+0000 7f1ca1663700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: pgmap v79: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: pgmap v80: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: pgmap v81: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded (PG_DEGRADED) 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[83585]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: pgmap v79: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: pgmap v80: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: pgmap v81: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded (PG_DEGRADED) 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:44.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:44 smithi017 ceph-mon[88777]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-14T17:46:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: pgmap v79: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: pgmap v80: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: pgmap v81: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded (PG_DEGRADED) 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:44.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:44 smithi111 ceph-mon[68099]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-02-14T17:46:45.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:45 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded) 2024-02-14T17:46:45.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:45 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded) 2024-02-14T17:46:45.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:45 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/627 objects degraded (4.785%), 9 pgs degraded) 2024-02-14T17:46:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:46 smithi111 ceph-mon[68099]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-02-14T17:46:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:46 smithi017 ceph-mon[83585]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-02-14T17:46:46.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:46 smithi017 ceph-mon[88777]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-02-14T17:46:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:48.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:48 smithi111 ceph-mon[68099]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:46:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:48 smithi017 ceph-mon[83585]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:46:48.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:48 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:48.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:48 smithi017 ceph-mon[88777]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-14T17:46:50.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:50 smithi111 ceph-mon[68099]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:50.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:50 smithi017 ceph-mon[83585]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:50.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:50 smithi017 ceph-mon[88777]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[83585]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-02-14T17:46:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:52 smithi017 ceph-mon[88777]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-02-14T17:46:52.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:46:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:46:51] "GET /metrics HTTP/1.1" 200 38163 "" "Prometheus/2.43.0" 2024-02-14T17:46:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:46:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:46:52.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:52 smithi111 ceph-mon[68099]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-02-14T17:46:54.172 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:46:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:54 smithi017 ceph-mon[83585]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:46:54.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:54 smithi017 ceph-mon[88777]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:46:54.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:54 smithi111 ceph-mon[68099]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:46:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:46:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (4m) 15s ago 26m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:46:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (4m) 2m ago 26m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:46:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 15s ago 25m 61.2M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:46:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (5m) 2m ago 30m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:46:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (15m) 15s ago 32m 502M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:46:54.836 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (2m) 15s ago 32m 49.6M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (3m) 2m ago 31m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (2m) 15s ago 31m 36.2M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (5m) 15s ago 26m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (5m) 2m ago 26m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (21s) 15s ago 30m 12.3M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (30m) 15s ago 30m 55.6M 2993M 17.2.0 e1d6a67b021e 8b90c6a1c7fa 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (29m) 15s ago 29m 52.5M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (29m) 15s ago 29m 53.4M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (29m) 2m ago 29m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (28m) 2m ago 28m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (28m) 2m ago 28m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (27m) 2m ago 27m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (4m) 2m ago 27m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:46:54.837 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (25m) 15s ago 25m 79.5M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:46:54.838 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (25m) 2m ago 25m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 1 2024-02-14T17:46:55.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 6 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:46:55.243 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:46:55.584 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:46:55.584 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:46:55.584 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:46:55.584 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-02-14T17:46:55.585 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:46:55.585 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "1/8 daemons upgraded", 2024-02-14T17:46:55.585 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:46:55.585 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:46:55.585 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:46:55.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[83585]: from='client.54217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[83585]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[83585]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.585 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/951909956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:55.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[88777]: from='client.54217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[88777]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[88777]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.586 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:55 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/951909956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:55 smithi111 ceph-mon[68099]: from='client.54217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:55 smithi111 ceph-mon[68099]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:55 smithi111 ceph-mon[68099]: from='client.44292 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:55.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:55 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/951909956' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:46:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:56 smithi017 ceph-mon[83585]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:56 smithi017 ceph-mon[83585]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:56 smithi017 ceph-mon[88777]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:56 smithi017 ceph-mon[88777]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:56 smithi111 ceph-mon[68099]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:46:56.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:56 smithi111 ceph-mon[68099]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:46:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:58 smithi017 ceph-mon[83585]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:58 smithi017 ceph-mon[88777]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:46:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:58 smithi111 ceph-mon[68099]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:46:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:59 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:59.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:46:59 smithi017 ceph-mon[83585]: Upgrade: osd.1 is safe to restart 2024-02-14T17:46:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:59 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:46:59 smithi017 ceph-mon[88777]: Upgrade: osd.1 is safe to restart 2024-02-14T17:46:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:59 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-02-14T17:46:59.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:46:59 smithi111 ceph-mon[68099]: Upgrade: osd.1 is safe to restart 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[83585]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[88777]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:47:00.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:00 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:00 smithi111 ceph-mon[68099]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:00 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:00 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-14T17:47:00.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:00 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:01 smithi017 ceph-mon[83585]: Upgrade: Updating osd.1 2024-02-14T17:47:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:01 smithi017 ceph-mon[83585]: Deploying daemon osd.1 on smithi017 2024-02-14T17:47:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:01 smithi017 ceph-mon[88777]: Upgrade: Updating osd.1 2024-02-14T17:47:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:01 smithi017 ceph-mon[88777]: Deploying daemon osd.1 on smithi017 2024-02-14T17:47:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:01 smithi111 ceph-mon[68099]: Upgrade: Updating osd.1 2024-02-14T17:47:01.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:01 smithi111 ceph-mon[68099]: Deploying daemon osd.1 on smithi017 2024-02-14T17:47:02.129 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:47:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:47:01] "GET /metrics HTTP/1.1" 200 38163 "" "Prometheus/2.43.0" 2024-02-14T17:47:02.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:02 smithi017 ceph-mon[83585]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:02.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:02 smithi017 ceph-mon[88777]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:02.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:02 smithi111 ceph-mon[68099]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:03.369 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:03 smithi017 systemd[1]: Stopping Ceph osd.1 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:47:03.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:03 smithi017 ceph-mon[83585]: osd.1 marked itself down and dead 2024-02-14T17:47:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:03 smithi017 ceph-mon[88777]: osd.1 marked itself down and dead 2024-02-14T17:47:03.690 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[35320]: 2024-02-14T17:47:03.375+0000 7f490e2da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:47:03.691 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[35320]: 2024-02-14T17:47:03.375+0000 7f490e2da700 -1 osd.1 116 *** Got signal Terminated *** 2024-02-14T17:47:03.691 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:03 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[35320]: 2024-02-14T17:47:03.375+0000 7f490e2da700 -1 osd.1 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:47:03.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:03 smithi111 ceph-mon[68099]: osd.1 marked itself down and dead 2024-02-14T17:47:04.093 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:03 smithi017 podman[100930]: 2024-02-14 17:47:03.805202895 +0000 UTC m=+0.550059403 container died 8b90c6a1c7fac74b5deb0f63b754ced83b3f1f54d63b05b417f797208dd82097 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, io.k8s.display-name=CentOS Stream 8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.openshift.expose-services=, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.openshift.tags=base centos centos-stream, architecture=x86_64, RELEASE=HEAD, io.buildah.version=1.19.8, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-02-14T17:47:04.435 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:04 smithi017 podman[100930]: 2024-02-14 17:47:04.191198054 +0000 UTC m=+0.936054561 container cleanup 8b90c6a1c7fac74b5deb0f63b754ced83b3f1f54d63b05b417f797208dd82097 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, release=754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, build-date=2022-05-03T08:36:31.336870, version=8, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., GIT_CLEAN=True) 2024-02-14T17:47:04.435 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:04 smithi017 bash[100930]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1 2024-02-14T17:47:04.690 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:04 smithi017 podman[100950]: 2024-02-14 17:47:04.442448084 +0000 UTC m=+0.637181796 container remove 8b90c6a1c7fac74b5deb0f63b754ced83b3f1f54d63b05b417f797208dd82097 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-14T17:47:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:47:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: osdmap e117: 8 total, 7 up, 8 in 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[88777]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: osdmap e117: 8 total, 7 up, 8 in 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.693 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:04 smithi017 ceph-mon[83585]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:47:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: osdmap e117: 8 total, 7 up, 8 in 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:04.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:04 smithi111 ceph-mon[68099]: failed to encode map e117 with expected crc 2024-02-14T17:47:05.072 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:04 smithi017 podman[101043]: 2024-02-14 17:47:04.670686957 +0000 UTC m=+0.018916657 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:47:05.072 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:04 smithi017 podman[101043]: 2024-02-14 17:47:04.785160105 +0000 UTC m=+0.133389792 container create f79c1ad85d769b035cff799e9046a558d0360161325937d8b79a5d31625bb6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:47:05.072 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:04 smithi017 podman[101043]: 2024-02-14 17:47:04.98817014 +0000 UTC m=+0.336399839 container init f79c1ad85d769b035cff799e9046a558d0360161325937d8b79a5d31625bb6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, io.buildah.version=1.33.5) 2024-02-14T17:47:05.072 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:04 smithi017 podman[101043]: 2024-02-14 17:47:04.991726285 +0000 UTC m=+0.339955977 container start f79c1ad85d769b035cff799e9046a558d0360161325937d8b79a5d31625bb6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-14T17:47:05.072 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:05 smithi017 podman[101043]: 2024-02-14 17:47:05.070857252 +0000 UTC m=+0.419086939 container attach f79c1ad85d769b035cff799e9046a558d0360161325937d8b79a5d31625bb6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-14T17:47:05.440 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:05 smithi017 podman[101043]: 2024-02-14 17:47:05.298977855 +0000 UTC m=+0.647207532 container died f79c1ad85d769b035cff799e9046a558d0360161325937d8b79a5d31625bb6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-14T17:47:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: osdmap e118: 8 total, 7 up, 8 in 2024-02-14T17:47:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:05 smithi111 ceph-mon[68099]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: osdmap e118: 8 total, 7 up, 8 in 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.884 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[83585]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: osdmap e118: 8 total, 7 up, 8 in 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.885 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.886 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.886 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.886 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.886 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:05.886 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:05 smithi017 ceph-mon[88777]: failed to encode map e118 with expected crc 2024-02-14T17:47:06.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:05 smithi017 podman[101069]: 2024-02-14 17:47:05.882848708 +0000 UTC m=+0.576284917 container remove f79c1ad85d769b035cff799e9046a558d0360161325937d8b79a5d31625bb6a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-14T17:47:06.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:05 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-14T17:47:06.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:05 smithi017 systemd[1]: Stopped Ceph osd.1 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:47:06.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:05 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.1.service: Consumed 15.159s CPU time. 2024-02-14T17:47:06.191 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:06 smithi017 systemd[1]: Starting Ceph osd.1 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:47:06.645 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:06 smithi017 ceph-mon[83585]: pgmap v94: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:47:06.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:06.646 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:06.646 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:06 smithi017 ceph-mon[88777]: pgmap v94: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:47:06.646 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:06.646 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:06.646 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:06 smithi017 podman[101174]: 2024-02-14 17:47:06.22923604 +0000 UTC m=+0.018821121 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:47:06.646 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:06 smithi017 podman[101174]: 2024-02-14 17:47:06.325733306 +0000 UTC m=+0.115318424 container create 56d6661fe45ce1ff442b9a855fae6840956192317a7a75a11ba3aee89da75075 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240214, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-02-14T17:47:06.646 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:06 smithi017 podman[101174]: 2024-02-14 17:47:06.561420131 +0000 UTC m=+0.351005204 container init 56d6661fe45ce1ff442b9a855fae6840956192317a7a75a11ba3aee89da75075 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-14T17:47:06.647 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:06 smithi017 podman[101174]: 2024-02-14 17:47:06.564713054 +0000 UTC m=+0.354298134 container start 56d6661fe45ce1ff442b9a855fae6840956192317a7a75a11ba3aee89da75075 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-14T17:47:06.647 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:06 smithi017 podman[101174]: 2024-02-14 17:47:06.644671652 +0000 UTC m=+0.434256729 container attach 56d6661fe45ce1ff442b9a855fae6840956192317a7a75a11ba3aee89da75075 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-14T17:47:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:06 smithi111 ceph-mon[68099]: pgmap v94: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:47:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:06.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:07.374 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate[101189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-14T17:47:07.374 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate[101189]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-14T17:47:07.374 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 bash[101174]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-14T17:47:07.374 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 bash[101174]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 bash[101174]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 bash[101174]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 bash[101174]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 bash[101174]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 bash[101174]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate[101189]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate[101189]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate[101189]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate[101189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate[101189]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-14T17:47:07.375 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 podman[101174]: 2024-02-14 17:47:07.032835413 +0000 UTC m=+0.822420487 container died 56d6661fe45ce1ff442b9a855fae6840956192317a7a75a11ba3aee89da75075 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:47:07.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 76/627 objects degraded (12.121%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:47:07.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:07 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 76/627 objects degraded (12.121%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:47:07.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:07 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 76/627 objects degraded (12.121%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:47:08.048 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 podman[101312]: 2024-02-14 17:47:07.740632229 +0000 UTC m=+0.699647028 container remove 56d6661fe45ce1ff442b9a855fae6840956192317a7a75a11ba3aee89da75075 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, io.buildah.version=1.33.5) 2024-02-14T17:47:08.048 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 podman[101378]: 2024-02-14 17:47:07.949783788 +0000 UTC m=+0.109807176 container create 8beec53ad36659ddbbd09cc045502f2460594b781b9c748bb055ecb86c636e33 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:47:08.048 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:07 smithi017 podman[101378]: 2024-02-14 17:47:07.857937561 +0000 UTC m=+0.017960892 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:47:08.440 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:08 smithi017 podman[101378]: 2024-02-14 17:47:08.143430278 +0000 UTC m=+0.303453602 container init 8beec53ad36659ddbbd09cc045502f2460594b781b9c748bb055ecb86c636e33 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-14T17:47:08.440 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:08 smithi017 podman[101378]: 2024-02-14 17:47:08.146600064 +0000 UTC m=+0.306623390 container start 8beec53ad36659ddbbd09cc045502f2460594b781b9c748bb055ecb86c636e33 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-14T17:47:08.440 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-osd[101397]: -- 172.21.15.17:0/2690821419 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b802dd11e0 con 0x55b802da3400 2024-02-14T17:47:08.440 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:08 smithi017 bash[101378]: 8beec53ad36659ddbbd09cc045502f2460594b781b9c748bb055ecb86c636e33 2024-02-14T17:47:08.440 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:08 smithi017 systemd[1]: Started Ceph osd.1 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:47:08.940 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[101393]: 2024-02-14T17:47:08.688+0000 7f4e0874a7c0 -1 Falling back to public interface 2024-02-14T17:47:08.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:08.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[88777]: pgmap v95: 161 pgs: 18 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-02-14T17:47:08.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:08.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:08.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[83585]: pgmap v95: 161 pgs: 18 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-02-14T17:47:08.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:08 smithi111 ceph-mon[68099]: pgmap v95: 161 pgs: 18 active+undersized+degraded, 1 stale+active+clean, 38 active+undersized, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 76/627 objects degraded (12.121%) 2024-02-14T17:47:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:08.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:10.861 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:10 smithi017 ceph-mon[83585]: pgmap v96: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:10.862 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:10 smithi017 ceph-mon[88777]: pgmap v96: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:10.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:10 smithi111 ceph-mon[68099]: pgmap v96: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:11.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-mon[83585]: from='osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:47:11.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-mon[83585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:47:11.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-mon[88777]: from='osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:47:11.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-mon[88777]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:47:11.690 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[101393]: 2024-02-14T17:47:11.412+0000 7f4e0874a7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-02-14T17:47:11.691 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[101393]: 2024-02-14T17:47:11.438+0000 7f4e0874a7c0 -1 osd.1 116 log_to_monitors true 2024-02-14T17:47:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:11 smithi111 ceph-mon[68099]: from='osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:47:11.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:11 smithi111 ceph-mon[68099]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-14T17:47:12.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[101393]: 2024-02-14T17:47:11.799+0000 7f4df5b9a700 -1 osd.1 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:47:12.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:47:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:47:11] "GET /metrics HTTP/1.1" 200 38160 "" "Prometheus/2.43.0" 2024-02-14T17:47:12.846 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: pgmap v97: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: osdmap e119: 8 total, 7 up, 8 in 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: from='osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[88777]: from='osd.1 ' entity='osd.1' 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: pgmap v97: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:12.847 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: osdmap e119: 8 total, 7 up, 8 in 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: from='osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:12 smithi017 ceph-mon[83585]: from='osd.1 ' entity='osd.1' 2024-02-14T17:47:12.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: pgmap v97: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:12.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-14T17:47:12.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: osdmap e119: 8 total, 7 up, 8 in 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: from='osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: failed to encode map e119 with expected crc 2024-02-14T17:47:12.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:12 smithi111 ceph-mon[68099]: from='osd.1 ' entity='osd.1' 2024-02-14T17:47:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:47:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: failed to encode map e120 with expected crc 2024-02-14T17:47:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: failed to encode map e120 with expected crc 2024-02-14T17:47:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098] boot 2024-02-14T17:47:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: osdmap e120: 8 total, 8 up, 8 in 2024-02-14T17:47:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:47:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: failed to encode map e120 with expected crc 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: failed to encode map e120 with expected crc 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098] boot 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: osdmap e120: 8 total, 8 up, 8 in 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:13.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:13 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:13.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:47:13.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: failed to encode map e120 with expected crc 2024-02-14T17:47:13.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: failed to encode map e120 with expected crc 2024-02-14T17:47:13.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: osd.1 [v2:172.21.15.17:6810/3782966098,v1:172.21.15.17:6811/3782966098] boot 2024-02-14T17:47:13.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: osdmap e120: 8 total, 8 up, 8 in 2024-02-14T17:47:13.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-14T17:47:13.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:13.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:13 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:14.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: pgmap v100: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:14.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.848 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: osdmap e121: 8 total, 8 up, 8 in 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: pgmap v100: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:14.849 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: osdmap e121: 8 total, 8 up, 8 in 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: failed to encode map e121 with expected crc 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:14.850 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:14 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:15.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: pgmap v100: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 85/627 objects degraded (13.557%) 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: osdmap e121: 8 total, 8 up, 8 in 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: failed to encode map e121 with expected crc 2024-02-14T17:47:15.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:15.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:14 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:16.135 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:15 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded (PG_DEGRADED) 2024-02-14T17:47:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:15 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded (PG_DEGRADED) 2024-02-14T17:47:16.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:15 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded (PG_DEGRADED) 2024-02-14T17:47:17.058 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:16 smithi017 ceph-mon[83585]: pgmap v102: 161 pgs: 6 peering, 19 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-14T17:47:17.059 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:16 smithi017 ceph-mon[88777]: pgmap v102: 161 pgs: 6 peering, 19 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-14T17:47:17.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:16 smithi111 ceph-mon[68099]: pgmap v102: 161 pgs: 6 peering, 19 active+undersized+degraded, 38 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 78/627 objects degraded (12.440%) 2024-02-14T17:47:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: pgmap v103: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: pgmap v104: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: pgmap v105: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: Upgrade: Finalizing container_image settings 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:47:18.692 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: pgmap v103: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: pgmap v104: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: pgmap v105: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.693 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: Upgrade: Finalizing container_image settings 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:47:18.694 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.695 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: Upgrade: Complete! 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: pgmap v106: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:18.696 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: pgmap v107: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.697 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: Upgrade: Complete! 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: pgmap v106: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:18.698 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: pgmap v107: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: pgmap v108: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: pgmap v109: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded) 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.699 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.700 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.700 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: pgmap v108: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-02-14T17:47:18.700 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: pgmap v109: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:18.700 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.700 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:18 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded) 2024-02-14T17:47:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: pgmap v103: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: pgmap v104: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: pgmap v105: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:47:18.735 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: Upgrade: Finalizing container_image settings 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:47:18.736 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.737 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: Upgrade: Complete! 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: pgmap v106: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: pgmap v107: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:18.738 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: pgmap v108: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s 2024-02-14T17:47:18.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: pgmap v109: 161 pgs: 6 peering, 155 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:18.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:18.739 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:18 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/627 objects degraded (12.440%), 19 pgs degraded) 2024-02-14T17:47:21.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:20 smithi017 ceph-mon[83585]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:21.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:20 smithi017 ceph-mon[88777]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:21.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:20 smithi111 ceph-mon[68099]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:22.028 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:22.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:22.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:22.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:22.028 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[83585]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:22.029 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:47:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:47:21] "GET /metrics HTTP/1.1" 200 38177 "" "Prometheus/2.43.0" 2024-02-14T17:47:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:22 smithi017 ceph-mon[88777]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:22 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:22 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:22 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:22.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:22 smithi111 ceph-mon[68099]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:47:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:24 smithi017 ceph-mon[83585]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-02-14T17:47:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:24 smithi017 ceph-mon[88777]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-02-14T17:47:25.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:24 smithi111 ceph-mon[68099]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-02-14T17:47:26.630 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:47:26.939 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:26 smithi017 ceph-mon[83585]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-02-14T17:47:26.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:26 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:26.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:26 smithi017 ceph-mon[88777]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-02-14T17:47:26.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:26 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:26 smithi111 ceph-mon[68099]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 0 op/s 2024-02-14T17:47:27.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:26 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:27.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:27 smithi017 ceph-mon[83585]: from='client.54253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:27.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:27 smithi017 ceph-mon[88777]: from='client.54253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:27 smithi111 ceph-mon[68099]: from='client.54253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:28.892 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:47:28.892 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (5m) 16s ago 27m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:47:28.892 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (5m) 2m ago 26m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:47:28.892 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 16s ago 26m 61.3M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:47:28.892 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (6m) 2m ago 31m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (16m) 16s ago 33m 504M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (3m) 16s ago 33m 49.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (3m) 2m ago 31m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (2m) 16s ago 31m 37.2M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (5m) 16s ago 27m 15.6M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (5m) 2m ago 27m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (55s) 16s ago 31m 42.8M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (21s) 16s ago 30m 14.9M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (30m) 16s ago 30m 54.6M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (30m) 16s ago 30m 54.5M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (29m) 2m ago 29m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (29m) 2m ago 29m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (28m) 2m ago 28m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (28m) 2m ago 28m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:47:28.893 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (5m) 2m ago 27m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:47:28.894 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (26m) 16s ago 26m 79.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:47:28.894 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (26m) 2m ago 26m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:47:29.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:29.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:28 smithi017 ceph-mon[88777]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:47:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:28 smithi017 ceph-mon[83585]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:47:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:28 smithi111 ceph-mon[68099]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:47:29.466 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-02-14T17:47:31.020 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:30 smithi017 ceph-mon[83585]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:31.020 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:30 smithi017 ceph-mon[83585]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:31.021 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:30 smithi017 ceph-mon[88777]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:31.021 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:30 smithi017 ceph-mon[88777]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:30 smithi111 ceph-mon[68099]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:30 smithi111 ceph-mon[68099]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:31.550 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:47:31.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:31 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1601940826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:31.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:31 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1601940826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:31.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:47:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:47:31] "GET /metrics HTTP/1.1" 200 38177 "" "Prometheus/2.43.0" 2024-02-14T17:47:32.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-02-14T17:47:32.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:31 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1601940826' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:33.087 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:32 smithi017 ceph-mon[83585]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:33.087 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:32 smithi017 ceph-mon[88777]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:33.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:32 smithi111 ceph-mon[68099]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:35.119 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:34 smithi017 ceph-mon[83585]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:35.120 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:34 smithi017 ceph-mon[88777]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:34 smithi111 ceph-mon[68099]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:36.094 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:47:36.094 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:35 smithi017 ceph-mon[88777]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:36.095 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:35 smithi017 ceph-mon[83585]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:35 smithi111 ceph-mon[68099]: from='client.44334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:36.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:47:36.911 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:36 smithi017 ceph-mon[83585]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:36.912 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:36.912 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:36 smithi017 ceph-mon[88777]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:36.912 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:36 smithi111 ceph-mon[68099]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": null, 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": false, 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "", 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": null, 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-02-14T17:47:38.517 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:47:38.518 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:47:39.162 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:39.163 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:38 smithi017 ceph-mon[83585]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:39.163 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:39.163 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:38 smithi017 ceph-mon[88777]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:39.189 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:47:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:38 smithi111 ceph-mon[68099]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:40.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:39 smithi017 ceph-mon[83585]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:40.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:39 smithi017 ceph-mon[88777]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:40.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:39 smithi111 ceph-mon[68099]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:40.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:40 smithi017 ceph-mon[83585]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:40.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:40 smithi017 ceph-mon[88777]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:41.188 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:47:41.189 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:47:41.189 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:47:41.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:40 smithi111 ceph-mon[68099]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:41.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-02-14T17:47:42.080 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:41 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/9108610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:47:42.080 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:41 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/9108610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:47:42.081 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:47:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:47:41] "GET /metrics HTTP/1.1" 200 38177 "" "Prometheus/2.43.0" 2024-02-14T17:47:42.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:41 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/9108610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:47:43.044 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:42 smithi017 ceph-mon[83585]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:43.045 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:42 smithi017 ceph-mon[88777]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:43.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:42 smithi111 ceph-mon[68099]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:44.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:44 smithi017 ceph-mon[83585]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:44.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:44 smithi017 ceph-mon[83585]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:44.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:44 smithi017 ceph-mon[88777]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:44.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:44 smithi017 ceph-mon[88777]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:45.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:44 smithi111 ceph-mon[68099]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:47:45.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:44 smithi111 ceph-mon[68099]: from='client.44352 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:45.812 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:47:46.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-14T17:47:47.006 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[88777]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:47.006 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[88777]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:47:47.006 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:47.006 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:47.006 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:47.006 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[83585]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[83585]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:47.007 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:46 smithi111 ceph-mon[68099]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:47:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:46 smithi111 ceph-mon[68099]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:47:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:47:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:48.090 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:47 smithi017 ceph-mon[83585]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-02-14T17:47:48.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:47 smithi017 ceph-mon[83585]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 588 B/s rd, 0 op/s 2024-02-14T17:47:48.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:47 smithi017 ceph-mon[83585]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:47:48.091 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:47 smithi017 ceph-mon[88777]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-02-14T17:47:48.091 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:47 smithi017 ceph-mon[88777]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 588 B/s rd, 0 op/s 2024-02-14T17:47:48.092 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:47 smithi017 ceph-mon[88777]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:47:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:47 smithi111 ceph-mon[68099]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-02-14T17:47:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:47 smithi111 ceph-mon[68099]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 588 B/s rd, 0 op/s 2024-02-14T17:47:48.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:47 smithi111 ceph-mon[68099]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:47:48.576 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:47:49.056 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:48 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:49.056 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:49.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (5m) 36s ago 27m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (5m) 3m ago 27m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 36s ago 26m 61.3M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (6m) 3m ago 31m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (16m) 36s ago 33m 504M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (3m) 36s ago 33m 49.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (4m) 3m ago 32m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:47:49.277 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (2m) 36s ago 32m 37.2M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (6m) 36s ago 27m 15.6M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (6m) 3m ago 27m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (75s) 36s ago 31m 42.8M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (41s) 36s ago 31m 14.9M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (30m) 36s ago 30m 54.6M 2993M 17.2.0 e1d6a67b021e a838da2fbc15 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (30m) 36s ago 30m 54.5M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (30m) 3m ago 30m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (29m) 3m ago 29m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (29m) 3m ago 29m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (28m) 3m ago 28m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (5m) 3m ago 28m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (26m) 36s ago 26m 79.7M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:47:49.278 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (26m) 3m ago 26m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:47:49.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 7 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:47:49.689 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:47:50.043 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:47:50.043 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:47:50.043 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:47:50.043 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-02-14T17:47:50.044 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:47:50.044 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "2/8 daemons upgraded", 2024-02-14T17:47:50.044 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:47:50.044 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:47:50.044 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:47:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/309310085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:50.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.193 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-14T17:47:50.193 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:49 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/309310085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:50.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='client.54295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:50.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-14T17:47:50.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:49 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/309310085' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:47:51.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:51.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:51.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:47:51.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:47:51.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mgr 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mon 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all crash 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[83585]: Upgrade: osd.2 is safe to restart 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mgr 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mon 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all crash 2024-02-14T17:47:51.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-14T17:47:51.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:50 smithi017 ceph-mon[88777]: Upgrade: osd.2 is safe to restart 2024-02-14T17:47:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:47:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:47:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mgr 2024-02-14T17:47:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mon 2024-02-14T17:47:51.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all crash 2024-02-14T17:47:51.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-02-14T17:47:51.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:50 smithi111 ceph-mon[68099]: Upgrade: osd.2 is safe to restart 2024-02-14T17:47:52.028 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:47:51] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[83585]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[83585]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[88777]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:52.029 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[88777]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:52.030 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:52.030 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:52.030 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:47:52.030 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:52.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:51 smithi111 ceph-mon[68099]: from='client.54319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:47:52.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:51 smithi111 ceph-mon[68099]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:52.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:47:52.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:47:52.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-14T17:47:52.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:47:52.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:52 smithi017 ceph-mon[83585]: Upgrade: Updating osd.2 2024-02-14T17:47:52.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:52 smithi017 ceph-mon[83585]: Deploying daemon osd.2 on smithi017 2024-02-14T17:47:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:52 smithi017 ceph-mon[88777]: Upgrade: Updating osd.2 2024-02-14T17:47:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:52 smithi017 ceph-mon[88777]: Deploying daemon osd.2 on smithi017 2024-02-14T17:47:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:52 smithi111 ceph-mon[68099]: Upgrade: Updating osd.2 2024-02-14T17:47:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:52 smithi111 ceph-mon[68099]: Deploying daemon osd.2 on smithi017 2024-02-14T17:47:53.862 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:53 smithi017 ceph-mon[83585]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:53.862 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:53 smithi017 ceph-mon[88777]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:54.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:53 smithi111 ceph-mon[68099]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:54.492 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:54 smithi017 systemd[1]: Stopping Ceph osd.2 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:47:54.841 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[38229]: 2024-02-14T17:47:54.489+0000 7f70d885c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:47:54.842 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[38229]: 2024-02-14T17:47:54.489+0000 7f70d885c700 -1 osd.2 121 *** Got signal Terminated *** 2024-02-14T17:47:54.842 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[38229]: 2024-02-14T17:47:54.489+0000 7f70d885c700 -1 osd.2 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:47:55.188 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:54 smithi017 ceph-mon[88777]: osd.2 marked itself down and dead 2024-02-14T17:47:55.188 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:54 smithi017 ceph-mon[83585]: osd.2 marked itself down and dead 2024-02-14T17:47:55.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:54 smithi111 ceph-mon[68099]: osd.2 marked itself down and dead 2024-02-14T17:47:55.440 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:55 smithi017 podman[107519]: 2024-02-14 17:47:55.187135323 +0000 UTC m=+0.820127148 container died a838da2fbc152734645b242d7b42c1b64426d18ea681e67b869853ca8d9666d4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , architecture=x86_64, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-02-14T17:47:55.792 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:55 smithi017 podman[107519]: 2024-02-14 17:47:55.614858885 +0000 UTC m=+1.247850707 container cleanup a838da2fbc152734645b242d7b42c1b64426d18ea681e67b869853ca8d9666d4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, name=centos-stream, ceph=True, vendor=Red Hat, Inc.) 2024-02-14T17:47:55.792 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:55 smithi017 bash[107519]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2 2024-02-14T17:47:56.190 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:55 smithi017 podman[107537]: 2024-02-14 17:47:55.791098328 +0000 UTC m=+0.593156653 container remove a838da2fbc152734645b242d7b42c1b64426d18ea681e67b869853ca8d9666d4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, ceph=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, maintainer=Guillaume Abrioux ) 2024-02-14T17:47:56.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:56.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:47:56.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[83585]: osdmap e122: 8 total, 7 up, 8 in 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:55 smithi017 ceph-mon[88777]: osdmap e122: 8 total, 7 up, 8 in 2024-02-14T17:47:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:47:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:47:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: failed to encode map e122 with expected crc 2024-02-14T17:47:56.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:55 smithi111 ceph-mon[68099]: osdmap e122: 8 total, 7 up, 8 in 2024-02-14T17:47:56.504 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:56 smithi017 podman[107631]: 2024-02-14 17:47:56.10825496 +0000 UTC m=+0.018777575 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:47:56.505 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:56 smithi017 podman[107631]: 2024-02-14 17:47:56.209397832 +0000 UTC m=+0.119920443 container create c3fa8fc4682da22f9034950fb528c0aa25faa4958bf97707603f985e11e5fd4d (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:47:56.505 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:56 smithi017 podman[107631]: 2024-02-14 17:47:56.403458919 +0000 UTC m=+0.313981537 container init c3fa8fc4682da22f9034950fb528c0aa25faa4958bf97707603f985e11e5fd4d (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) 2024-02-14T17:47:56.505 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:56 smithi017 podman[107631]: 2024-02-14 17:47:56.406450758 +0000 UTC m=+0.316973377 container start c3fa8fc4682da22f9034950fb528c0aa25faa4958bf97707603f985e11e5fd4d (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:47:56.853 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:56 smithi017 podman[107631]: 2024-02-14 17:47:56.503352396 +0000 UTC m=+0.413875013 container attach c3fa8fc4682da22f9034950fb528c0aa25faa4958bf97707603f985e11e5fd4d (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:47:56.853 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:56 smithi017 podman[107631]: 2024-02-14 17:47:56.702321858 +0000 UTC m=+0.612844475 container died c3fa8fc4682da22f9034950fb528c0aa25faa4958bf97707603f985e11e5fd4d (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5) 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: osdmap e123: 8 total, 7 up, 8 in 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.118 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[83585]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: osdmap e123: 8 total, 7 up, 8 in 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.119 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:56 smithi017 ceph-mon[88777]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: osdmap e123: 8 total, 7 up, 8 in 2024-02-14T17:47:57.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:56 smithi111 ceph-mon[68099]: failed to encode map e123 with expected crc 2024-02-14T17:47:57.440 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 podman[107657]: 2024-02-14 17:47:57.223804822 +0000 UTC m=+0.516653652 container remove c3fa8fc4682da22f9034950fb528c0aa25faa4958bf97707603f985e11e5fd4d (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD) 2024-02-14T17:47:57.440 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-14T17:47:57.440 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 systemd[1]: Stopped Ceph osd.2 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:47:57.440 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.2.service: Consumed 12.801s CPU time. 2024-02-14T17:47:57.786 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 systemd[1]: Starting Ceph osd.2 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:47:57.786 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 podman[107760]: 2024-02-14 17:47:57.567375733 +0000 UTC m=+0.018429055 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:47:57.786 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 podman[107760]: 2024-02-14 17:47:57.667005552 +0000 UTC m=+0.118058913 container create 1c5339c9dcb67f8f4334c54d5fabeb4fb45cb78170b5e7aa6e2a0404e7018d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-14T17:47:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:57 smithi017 ceph-mon[83585]: pgmap v132: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:47:58.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:57 smithi017 ceph-mon[88777]: pgmap v132: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:47:58.190 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 podman[107760]: 2024-02-14 17:47:57.90289533 +0000 UTC m=+0.353948646 container init 1c5339c9dcb67f8f4334c54d5fabeb4fb45cb78170b5e7aa6e2a0404e7018d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-14T17:47:58.190 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:57 smithi017 podman[107760]: 2024-02-14 17:47:57.906303748 +0000 UTC m=+0.357357072 container start 1c5339c9dcb67f8f4334c54d5fabeb4fb45cb78170b5e7aa6e2a0404e7018d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-14T17:47:58.190 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 podman[107760]: 2024-02-14 17:47:58.002989097 +0000 UTC m=+0.454042416 container attach 1c5339c9dcb67f8f4334c54d5fabeb4fb45cb78170b5e7aa6e2a0404e7018d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-02-14T17:47:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:57 smithi111 ceph-mon[68099]: pgmap v132: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-14T17:47:58.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate[107775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-14T17:47:58.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate[107775]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-14T17:47:58.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 bash[107760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-14T17:47:58.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 bash[107760]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-14T17:47:58.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 bash[107760]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-14T17:47:58.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 bash[107760]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-14T17:47:58.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 bash[107760]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 bash[107760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 bash[107760]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate[107775]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate[107775]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate[107775]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate[107775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate[107775]: --> ceph-volume raw activate successful for osd ID: 2 2024-02-14T17:47:58.691 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 podman[107760]: 2024-02-14 17:47:58.377851879 +0000 UTC m=+0.828905195 container died 1c5339c9dcb67f8f4334c54d5fabeb4fb45cb78170b5e7aa6e2a0404e7018d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-14T17:47:58.999 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:58.999 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:59.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:47:59.336 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:58 smithi017 podman[107904]: 2024-02-14 17:47:58.998385477 +0000 UTC m=+0.611965774 container remove 1c5339c9dcb67f8f4334c54d5fabeb4fb45cb78170b5e7aa6e2a0404e7018d31 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-14T17:47:59.336 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:59 smithi017 podman[107968]: 2024-02-14 17:47:59.116975033 +0000 UTC m=+0.017910063 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:47:59.336 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:59 smithi017 podman[107968]: 2024-02-14 17:47:59.232531984 +0000 UTC m=+0.133466995 container create 89cf7ada9e899bd189de1b4cd457bb1619a8d9b38f9689aa1fc2ebef5930327c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214) 2024-02-14T17:47:59.861 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:59 smithi017 podman[107968]: 2024-02-14 17:47:59.448891902 +0000 UTC m=+0.349826913 container init 89cf7ada9e899bd189de1b4cd457bb1619a8d9b38f9689aa1fc2ebef5930327c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-14T17:47:59.861 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:59 smithi017 podman[107968]: 2024-02-14 17:47:59.452002095 +0000 UTC m=+0.352937107 container start 89cf7ada9e899bd189de1b4cd457bb1619a8d9b38f9689aa1fc2ebef5930327c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:47:59.861 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-osd[107986]: -- 172.21.15.17:0/2458666907 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b1b4a451e0 con 0x55b1b4a16c00 2024-02-14T17:47:59.861 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:59 smithi017 bash[107968]: 89cf7ada9e899bd189de1b4cd457bb1619a8d9b38f9689aa1fc2ebef5930327c 2024-02-14T17:47:59.861 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:47:59 smithi017 systemd[1]: Started Ceph osd.2 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:48:00.190 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:48:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[107982]: 2024-02-14T17:47:59.999+0000 7fd43f9997c0 -1 Falling back to public interface 2024-02-14T17:48:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[83585]: pgmap v133: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-02-14T17:48:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:00.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[88777]: pgmap v133: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:00.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-02-14T17:48:00.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:47:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:59 smithi111 ceph-mon[68099]: pgmap v133: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:59 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded (PG_DEGRADED) 2024-02-14T17:48:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:47:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:01.863 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:48:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:48:01] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-02-14T17:48:02.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:01 smithi017 ceph-mon[88777]: pgmap v134: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-02-14T17:48:02.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:01 smithi017 ceph-mon[83585]: pgmap v134: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-02-14T17:48:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:01 smithi111 ceph-mon[68099]: pgmap v134: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 58/627 objects degraded (9.250%) 2024-02-14T17:48:02.826 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:48:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[107982]: 2024-02-14T17:48:02.744+0000 7fd43f9997c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-02-14T17:48:02.827 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:48:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[107982]: 2024-02-14T17:48:02.768+0000 7fd43f9997c0 -1 osd.2 121 log_to_monitors true 2024-02-14T17:48:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:02 smithi017 ceph-mon[83585]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:48:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:02 smithi017 ceph-mon[88777]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:48:03.190 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:48:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[107982]: 2024-02-14T17:48:02.895+0000 7fd430df1700 -1 osd.2 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:48:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:02 smithi111 ceph-mon[68099]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[88777]: pgmap v135: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[88777]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[88777]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[88777]: osdmap e124: 8 total, 7 up, 8 in 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[88777]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[88777]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[88777]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[83585]: pgmap v135: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:04.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[83585]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:48:04.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[83585]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[83585]: osdmap e124: 8 total, 7 up, 8 in 2024-02-14T17:48:04.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[83585]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:48:04.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[83585]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:03 smithi017 ceph-mon[83585]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:03 smithi111 ceph-mon[68099]: pgmap v135: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:03 smithi111 ceph-mon[68099]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-14T17:48:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:03 smithi111 ceph-mon[68099]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:03 smithi111 ceph-mon[68099]: osdmap e124: 8 total, 7 up, 8 in 2024-02-14T17:48:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:03 smithi111 ceph-mon[68099]: from='osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:48:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:03 smithi111 ceph-mon[68099]: failed to encode map e124 with expected crc 2024-02-14T17:48:04.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:03 smithi111 ceph-mon[68099]: failed to encode map e124 with expected crc 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795] boot 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: osdmap e125: 8 total, 8 up, 8 in 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: pgmap v138: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[83585]: osdmap e126: 8 total, 8 up, 8 in 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795] boot 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: osdmap e125: 8 total, 8 up, 8 in 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: pgmap v138: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:05.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:04 smithi017 ceph-mon[88777]: osdmap e126: 8 total, 8 up, 8 in 2024-02-14T17:48:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:48:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: osd.2 [v2:172.21.15.17:6818/645042795,v1:172.21.15.17:6819/645042795] boot 2024-02-14T17:48:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: osdmap e125: 8 total, 8 up, 8 in 2024-02-14T17:48:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-14T17:48:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: failed to encode map e125 with expected crc 2024-02-14T17:48:05.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:05.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:05.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: pgmap v138: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:05.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:04 smithi111 ceph-mon[68099]: osdmap e126: 8 total, 8 up, 8 in 2024-02-14T17:48:06.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.169 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.170 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.171 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.171 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.171 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:06.171 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:05 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: failed to encode map e126 with expected crc 2024-02-14T17:48:06.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:06.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:05 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:07.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:07.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:07.331 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:07 smithi017 ceph-mon[83585]: pgmap v140: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:07.332 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:07.332 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:07.332 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:07 smithi017 ceph-mon[88777]: pgmap v140: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:07 smithi111 ceph-mon[68099]: pgmap v140: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-02-14T17:48:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[83585]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:09.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[88777]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:09.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:09.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:09.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:09 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:09 smithi111 ceph-mon[68099]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:09 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:09 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:09 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:09.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:09 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:09.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:09 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-02-14T17:48:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s 2024-02-14T17:48:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded) 2024-02-14T17:48:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: Upgrade: Finalizing container_image settings 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:48:10.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded) 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: Upgrade: Finalizing container_image settings 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:48:10.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:48:10.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: Upgrade: Complete! 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: Upgrade: Complete! 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-14T17:48:10.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:48:10.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:48:10.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:10 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 843 B/s rd, 0 op/s 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1009 B/s rd, 0 op/s 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 58/627 objects degraded (9.250%), 15 pgs degraded) 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: Upgrade: Finalizing container_image settings 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: Upgrade: Complete! 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:48:10.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:10 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:12.073 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:48:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:48:11] "GET /metrics HTTP/1.1" 200 38036 "" "Prometheus/2.43.0" 2024-02-14T17:48:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:12 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:12 smithi017 ceph-mon[83585]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:48:12.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:12 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:12.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:12 smithi017 ceph-mon[88777]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:48:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:12 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:12 smithi111 ceph-mon[68099]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:48:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:14 smithi017 ceph-mon[83585]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:14 smithi017 ceph-mon[88777]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:14 smithi111 ceph-mon[68099]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:16 smithi017 ceph-mon[83585]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-02-14T17:48:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:16 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:16 smithi017 ceph-mon[88777]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-02-14T17:48:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:16 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:16 smithi111 ceph-mon[68099]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-02-14T17:48:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:16 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:18.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:18.537 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:18 smithi111 ceph-mon[68099]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:18 smithi017 ceph-mon[88777]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:18 smithi017 ceph-mon[83585]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:20 smithi017 ceph-mon[83585]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:48:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:20 smithi017 ceph-mon[88777]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:48:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:20 smithi111 ceph-mon[68099]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:48:21.138 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:48:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:21 smithi017 ceph-mon[83585]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:21 smithi017 ceph-mon[88777]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:21.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:21.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:21.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:21 smithi111 ceph-mon[68099]: from='client.54331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:21.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:21.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:22.055 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:48:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:48:21] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-02-14T17:48:22.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:22 smithi017 ceph-mon[83585]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:22.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:22 smithi017 ceph-mon[88777]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:22.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:22 smithi111 ceph-mon[68099]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:23.223 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:48:23.223 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (6m) 18s ago 28m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:48:23.223 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (5m) 3m ago 27m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:48:23.223 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 18s ago 27m 61.3M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:48:23.223 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (7m) 3m ago 32m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (17m) 18s ago 33m 506M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (3m) 18s ago 34m 52.2M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (4m) 3m ago 32m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (3m) 18s ago 32m 39.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (6m) 18s ago 28m 15.2M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (6m) 3m ago 28m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (109s) 18s ago 32m 43.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (75s) 18s ago 31m 51.8M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (24s) 18s ago 31m 12.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (30m) 18s ago 30m 56.5M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (30m) 3m ago 30m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (30m) 3m ago 30m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (29m) 3m ago 29m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (29m) 3m ago 29m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:48:23.224 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (6m) 3m ago 28m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:48:23.225 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (27m) 18s ago 27m 80.0M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:48:23.225 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (27m) 3m ago 27m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:48:23.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-02-14T17:48:24.575 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:24 smithi017 ceph-mon[88777]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:24.575 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:24 smithi017 ceph-mon[88777]: from='client.44400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:24.575 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:24 smithi017 ceph-mon[83585]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:24.575 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:24 smithi017 ceph-mon[83585]: from='client.44400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:24 smithi111 ceph-mon[68099]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:24.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:24 smithi111 ceph-mon[68099]: from='client.44400 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:26.223 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:48:26.518 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:26 smithi017 ceph-mon[83585]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:26.518 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:26 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/44378666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:26.518 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:26 smithi017 ceph-mon[88777]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:26.519 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:26 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/44378666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:26 smithi111 ceph-mon[68099]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:26.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:26 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/44378666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:26.843 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-02-14T17:48:28.520 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:28.520 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:28 smithi017 ceph-mon[88777]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:28.521 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:28.521 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:28 smithi017 ceph-mon[83585]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:28.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:28 smithi111 ceph-mon[68099]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:30 smithi017 ceph-mon[88777]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:30.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:30 smithi017 ceph-mon[88777]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:30 smithi017 ceph-mon[83585]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:30.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:30 smithi017 ceph-mon[83585]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:30 smithi111 ceph-mon[68099]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:30.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:30 smithi111 ceph-mon[68099]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:31.152 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:48:31.893 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:48:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:48:31] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-02-14T17:48:31.924 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:48:32.497 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:32 smithi017 ceph-mon[83585]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:32.497 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:32 smithi017 ceph-mon[88777]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:32.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:32 smithi111 ceph-mon[68099]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:33.899 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:48:33.899 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": null, 2024-02-14T17:48:33.899 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": false, 2024-02-14T17:48:33.899 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "", 2024-02-14T17:48:33.899 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:48:33.900 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": null, 2024-02-14T17:48:33.900 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-02-14T17:48:33.900 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:48:33.900 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:48:34.528 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:34 smithi017 ceph-mon[83585]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:34.528 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:34 smithi017 ceph-mon[88777]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:34.547 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:48:34.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:34 smithi111 ceph-mon[68099]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:35.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:35 smithi017 ceph-mon[83585]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:35 smithi017 ceph-mon[88777]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:35.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:35 smithi111 ceph-mon[68099]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:36 smithi017 ceph-mon[83585]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:36.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:36 smithi017 ceph-mon[88777]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:36.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:36.628 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:48:36.628 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:48:36.628 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:48:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:36 smithi111 ceph-mon[68099]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:36.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:37.336 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:37 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2142314859' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:48:37.336 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:37 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2142314859' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:48:37.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-02-14T17:48:37.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:37 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2142314859' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:48:38.629 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:38.629 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:38 smithi017 ceph-mon[83585]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:38.629 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:38.630 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:38 smithi017 ceph-mon[88777]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:38.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:38 smithi111 ceph-mon[68099]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:40.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:40 smithi017 ceph-mon[83585]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:40.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:40 smithi017 ceph-mon[83585]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:40.602 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:40 smithi017 ceph-mon[88777]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:40.602 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:40 smithi017 ceph-mon[88777]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:40 smithi111 ceph-mon[68099]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:40.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:40 smithi111 ceph-mon[68099]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:41.568 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:48:41.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:48:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:48:41] "GET /metrics HTTP/1.1" 200 38183 "" "Prometheus/2.43.0" 2024-02-14T17:48:42.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-14T17:48:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:48:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[83585]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:42.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-02-14T17:48:42.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-02-14T17:48:42.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:42.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:42 smithi017 ceph-mon[88777]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:48:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:48:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:48:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:42.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:48:42.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-02-14T17:48:42.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-02-14T17:48:42.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:42.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:42 smithi111 ceph-mon[68099]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:48:44.491 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:48:44.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:44 smithi017 ceph-mon[83585]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:44.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:44 smithi017 ceph-mon[88777]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:44.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:44 smithi111 ceph-mon[68099]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (6m) 40s ago 28m 25.9M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (6m) 4m ago 28m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 40s ago 27m 61.3M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (7m) 4m ago 32m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (17m) 40s ago 34m 506M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (4m) 40s ago 34m 52.2M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:48:45.165 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (5m) 4m ago 32m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (3m) 40s ago 33m 39.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (7m) 40s ago 28m 15.2M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (7m) 4m ago 28m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (2m) 40s ago 32m 43.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (97s) 40s ago 32m 51.8M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (46s) 40s ago 31m 12.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (31m) 40s ago 31m 56.5M 2993M 17.2.0 e1d6a67b021e 214a4a052903 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (30m) 4m ago 30m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (30m) 4m ago 30m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (30m) 4m ago 30m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (29m) 4m ago 29m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (6m) 4m ago 29m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (27m) 40s ago 27m 80.0M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:48:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (27m) 4m ago 27m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:48:45.594 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 8 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:48:45.595 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:48:45.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='client.44436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:45.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:45.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/3276025309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='client.44436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:45.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-14T17:48:45.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:45 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/3276025309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.951 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:48:45.951 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:48:45.951 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:48:45.951 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-14T17:48:45.952 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:48:45.952 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "3/8 daemons upgraded", 2024-02-14T17:48:45.952 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:48:45.952 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:48:45.952 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='client.44436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-14T17:48:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:45 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/3276025309' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: from='client.54385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mgr 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mon 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all crash 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: Upgrade: osd.3 is safe to restart 2024-02-14T17:48:46.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[83585]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: from='client.54385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mgr 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mon 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all crash 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: Upgrade: osd.3 is safe to restart 2024-02-14T17:48:46.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:46 smithi017 ceph-mon[88777]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-02-14T17:48:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: from='client.54385 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:48:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:48:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mgr 2024-02-14T17:48:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mon 2024-02-14T17:48:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all crash 2024-02-14T17:48:46.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-02-14T17:48:46.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: Upgrade: osd.3 is safe to restart 2024-02-14T17:48:46.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:46 smithi111 ceph-mon[68099]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 612 B/s rd, 0 op/s 2024-02-14T17:48:47.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[83585]: from='client.44460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:47.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:47.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:48:47.731 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:47.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[88777]: from='client.44460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:47.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:47.731 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:48:47.732 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:47 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:47.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:47 smithi111 ceph-mon[68099]: from='client.44460 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:48:47.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:47 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:47.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:47 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-14T17:48:47.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:47 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:48:48.859 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[88777]: Upgrade: Updating osd.3 2024-02-14T17:48:48.859 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[88777]: Deploying daemon osd.3 on smithi017 2024-02-14T17:48:48.859 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:48.859 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[88777]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:48.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[83585]: Upgrade: Updating osd.3 2024-02-14T17:48:48.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[83585]: Deploying daemon osd.3 on smithi017 2024-02-14T17:48:48.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:48.860 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:48 smithi017 ceph-mon[83585]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:48 smithi111 ceph-mon[68099]: Upgrade: Updating osd.3 2024-02-14T17:48:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:48 smithi111 ceph-mon[68099]: Deploying daemon osd.3 on smithi017 2024-02-14T17:48:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:48 smithi111 ceph-mon[68099]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:50.409 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:50 smithi017 systemd[1]: Stopping Ceph osd.3 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:48:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:50 smithi017 ceph-mon[83585]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:50 smithi017 ceph-mon[83585]: osd.3 marked itself down and dead 2024-02-14T17:48:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:50 smithi017 ceph-mon[88777]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:50 smithi017 ceph-mon[88777]: osd.3 marked itself down and dead 2024-02-14T17:48:50.690 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:50 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[41194]: 2024-02-14T17:48:50.407+0000 7f04dba49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:48:50.691 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:50 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[41194]: 2024-02-14T17:48:50.407+0000 7f04dba49700 -1 osd.3 126 *** Got signal Terminated *** 2024-02-14T17:48:50.691 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:50 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[41194]: 2024-02-14T17:48:50.407+0000 7f04dba49700 -1 osd.3 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:48:50.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:50 smithi111 ceph-mon[68099]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:48:50.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:50 smithi111 ceph-mon[68099]: osd.3 marked itself down and dead 2024-02-14T17:48:51.329 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:51 smithi017 podman[114115]: 2024-02-14 17:48:51.061383541 +0000 UTC m=+0.762337095 container died 214a4a052903300797d38b4d4b19b174a63462c9e3f240bff16dd41e7700bdb3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, version=8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container) 2024-02-14T17:48:51.651 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:51 smithi017 podman[114115]: 2024-02-14 17:48:51.415667623 +0000 UTC m=+1.116621168 container cleanup 214a4a052903300797d38b4d4b19b174a63462c9e3f240bff16dd41e7700bdb3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, RELEASE=HEAD, vcs-type=git, distribution-scope=public, name=centos-stream, version=8, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-14T17:48:51.652 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:51 smithi017 bash[114115]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3 2024-02-14T17:48:51.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:51 smithi017 podman[114132]: 2024-02-14 17:48:51.775140345 +0000 UTC m=+0.714047767 container remove 214a4a052903300797d38b4d4b19b174a63462c9e3f240bff16dd41e7700bdb3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-type=git, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.component=centos-stream-container, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-02-14T17:48:51.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:48:51] "GET /metrics HTTP/1.1" 200 38179 "" "Prometheus/2.43.0" 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: osdmap e127: 8 total, 7 up, 8 in 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[88777]: osdmap e128: 8 total, 7 up, 8 in 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: osdmap e127: 8 total, 7 up, 8 in 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:51.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:51.943 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:51 smithi017 ceph-mon[83585]: osdmap e128: 8 total, 7 up, 8 in 2024-02-14T17:48:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:48:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: osdmap e127: 8 total, 7 up, 8 in 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: failed to encode map e127 with expected crc 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:48:51.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:51 smithi111 ceph-mon[68099]: osdmap e128: 8 total, 7 up, 8 in 2024-02-14T17:48:52.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:52 smithi017 podman[114225]: 2024-02-14 17:48:52.159667185 +0000 UTC m=+0.086506639 container create 40f2934585b51fb171a4d47c3562d8b0a97a1b4e7706a397b322b93bacf0e2a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD) 2024-02-14T17:48:52.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:52 smithi017 podman[114225]: 2024-02-14 17:48:52.090077668 +0000 UTC m=+0.016917130 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:48:52.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:52 smithi017 podman[114225]: 2024-02-14 17:48:52.370718572 +0000 UTC m=+0.297558038 container init 40f2934585b51fb171a4d47c3562d8b0a97a1b4e7706a397b322b93bacf0e2a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-14T17:48:52.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:52 smithi017 podman[114225]: 2024-02-14 17:48:52.373941716 +0000 UTC m=+0.300781182 container start 40f2934585b51fb171a4d47c3562d8b0a97a1b4e7706a397b322b93bacf0e2a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-14T17:48:52.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:52 smithi017 podman[114225]: 2024-02-14 17:48:52.470359725 +0000 UTC m=+0.397199211 container attach 40f2934585b51fb171a4d47c3562d8b0a97a1b4e7706a397b322b93bacf0e2a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:48:52.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:52 smithi017 podman[114225]: 2024-02-14 17:48:52.678406103 +0000 UTC m=+0.605245573 container died 40f2934585b51fb171a4d47c3562d8b0a97a1b4e7706a397b322b93bacf0e2a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True) 2024-02-14T17:48:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: pgmap v171: 161 pgs: 14 peering, 21 stale+active+clean, 126 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-14T17:48:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[88777]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: pgmap v171: 161 pgs: 14 peering, 21 stale+active+clean, 126 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:52 smithi017 ceph-mon[83585]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-14T17:48:52.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: pgmap v171: 161 pgs: 14 peering, 21 stale+active+clean, 126 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-14T17:48:52.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: failed to encode map e128 with expected crc 2024-02-14T17:48:52.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:52 smithi111 ceph-mon[68099]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-14T17:48:53.690 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:53 smithi017 podman[114251]: 2024-02-14 17:48:53.432412186 +0000 UTC m=+0.749278204 container remove 40f2934585b51fb171a4d47c3562d8b0a97a1b4e7706a397b322b93bacf0e2a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-14T17:48:53.690 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:53 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.3.service: Deactivated successfully. 2024-02-14T17:48:53.690 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:53 smithi017 systemd[1]: Stopped Ceph osd.3 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:48:53.690 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:53 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.3.service: Consumed 17.090s CPU time. 2024-02-14T17:48:53.691 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:53 smithi017 systemd[1]: Starting Ceph osd.3 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:48:53.959 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:53 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:48:53.960 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:53 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:48:53.960 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:53 smithi017 podman[114352]: 2024-02-14 17:48:53.85060738 +0000 UTC m=+0.095111638 container create 532015c3b9f725b54fcc5752bec048e4574fdc016e0da00b11fbb92461615dae (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-14T17:48:53.960 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:53 smithi017 podman[114352]: 2024-02-14 17:48:53.773518773 +0000 UTC m=+0.018022992 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:48:53.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:53 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:48:54.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 podman[114352]: 2024-02-14 17:48:54.086753254 +0000 UTC m=+0.331257471 container init 532015c3b9f725b54fcc5752bec048e4574fdc016e0da00b11fbb92461615dae (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-14T17:48:54.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 podman[114352]: 2024-02-14 17:48:54.090280153 +0000 UTC m=+0.334784379 container start 532015c3b9f725b54fcc5752bec048e4574fdc016e0da00b11fbb92461615dae (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-14T17:48:54.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 podman[114352]: 2024-02-14 17:48:54.169858129 +0000 UTC m=+0.414362358 container attach 532015c3b9f725b54fcc5752bec048e4574fdc016e0da00b11fbb92461615dae (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214) 2024-02-14T17:48:54.926 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-mon[83585]: pgmap v173: 161 pgs: 18 active+undersized+degraded, 14 peering, 34 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-14T17:48:54.926 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-mon[88777]: pgmap v173: 161 pgs: 18 active+undersized+degraded, 14 peering, 34 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-14T17:48:54.926 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate[114367]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate[114367]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 bash[114352]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 bash[114352]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 bash[114352]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 bash[114352]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 bash[114352]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 bash[114352]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 bash[114352]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate[114367]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate[114367]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate[114367]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate[114367]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate[114367]: --> ceph-volume raw activate successful for osd ID: 3 2024-02-14T17:48:54.927 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:54 smithi017 podman[114352]: 2024-02-14 17:48:54.565165895 +0000 UTC m=+0.809670112 container died 532015c3b9f725b54fcc5752bec048e4574fdc016e0da00b11fbb92461615dae (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-02-14T17:48:54.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:54 smithi111 ceph-mon[68099]: pgmap v173: 161 pgs: 18 active+undersized+degraded, 14 peering, 34 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-14T17:48:55.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 podman[114486]: 2024-02-14 17:48:55.181876407 +0000 UTC m=+0.611251487 container remove 532015c3b9f725b54fcc5752bec048e4574fdc016e0da00b11fbb92461615dae (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:48:55.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 podman[114552]: 2024-02-14 17:48:55.298959105 +0000 UTC m=+0.017934220 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:48:55.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 podman[114552]: 2024-02-14 17:48:55.399336504 +0000 UTC m=+0.118311614 container create f3a8d5491fd3ca0f52d515160fa0460ea516d60373b9e3705896538f3f6c6b1c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:48:55.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 podman[114552]: 2024-02-14 17:48:55.618758445 +0000 UTC m=+0.337733556 container init f3a8d5491fd3ca0f52d515160fa0460ea516d60373b9e3705896538f3f6c6b1c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:48:55.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 podman[114552]: 2024-02-14 17:48:55.621684259 +0000 UTC m=+0.340659377 container start f3a8d5491fd3ca0f52d515160fa0460ea516d60373b9e3705896538f3f6c6b1c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5) 2024-02-14T17:48:55.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 ceph-osd[114572]: -- 172.21.15.17:0/2267486438 <== mon.0 v2:172.21.15.17:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562f0f0871e0 con 0x562f0f059400 2024-02-14T17:48:55.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 bash[114552]: f3a8d5491fd3ca0f52d515160fa0460ea516d60373b9e3705896538f3f6c6b1c 2024-02-14T17:48:55.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:55 smithi017 systemd[1]: Started Ceph osd.3 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:48:56.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:48:56.162+0000 7f6b1b32c7c0 -1 Falling back to public interface 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[83585]: pgmap v174: 161 pgs: 18 active+undersized+degraded, 14 peering, 34 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[88777]: pgmap v174: 161 pgs: 18 active+undersized+degraded, 14 peering, 34 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:56.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:56 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:56.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:56 smithi111 ceph-mon[68099]: pgmap v174: 161 pgs: 18 active+undersized+degraded, 14 peering, 34 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-02-14T17:48:56.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:56 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:56.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:56 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:48:56.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:56 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:48:58.153 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:57 smithi017 ceph-mon[83585]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-02-14T17:48:58.153 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:57 smithi017 ceph-mon[88777]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-02-14T17:48:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:57 smithi111 ceph-mon[68099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 6 pgs peering) 2024-02-14T17:48:59.057 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:59.057 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:58 smithi017 ceph-mon[83585]: pgmap v175: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-14T17:48:59.057 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:48:58.895+0000 7f6b1b32c7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-02-14T17:48:59.058 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:48:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:48:58.921+0000 7f6b1b32c7c0 -1 osd.3 126 log_to_monitors true 2024-02-14T17:48:59.058 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:59.058 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:58 smithi017 ceph-mon[88777]: pgmap v175: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-14T17:48:59.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:48:59.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:58 smithi111 ceph-mon[68099]: pgmap v175: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-14T17:48:59.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:59 smithi017 ceph-mon[83585]: from='osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:48:59.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:48:59 smithi017 ceph-mon[83585]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:48:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:59 smithi017 ceph-mon[88777]: from='osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:48:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:48:59 smithi017 ceph-mon[88777]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:49:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:59 smithi111 ceph-mon[68099]: from='osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:49:00.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:48:59 smithi111 ceph-mon[68099]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-14T17:49:00.439 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:49:00.071+0000 7f6b0877c700 -1 osd.3 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: pgmap v176: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: osdmap e129: 8 total, 7 up, 8 in 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: from='osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: from='osd.3 ' entity='osd.3' 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[88777]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: pgmap v176: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: osdmap e129: 8 total, 7 up, 8 in 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: from='osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: from='osd.3 ' entity='osd.3' 2024-02-14T17:49:01.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:01.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:01.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.192 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:00 smithi017 ceph-mon[83585]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: pgmap v176: 161 pgs: 23 active+undersized+degraded, 43 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 77/627 objects degraded (12.281%) 2024-02-14T17:49:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-14T17:49:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: osdmap e129: 8 total, 7 up, 8 in 2024-02-14T17:49:01.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: from='osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi017", "root=default"]}]: dispatch 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: from='osd.3 ' entity='osd.3' 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: failed to encode map e129 with expected crc 2024-02-14T17:49:01.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:00 smithi111 ceph-mon[68099]: failed to encode map e129 with expected crc 2024-02-14T17:49:02.066 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:49:01] "GET /metrics HTTP/1.1" 200 38179 "" "Prometheus/2.43.0" 2024-02-14T17:49:02.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:02.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.066 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986] boot 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: osdmap e130: 8 total, 8 up, 8 in 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.067 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986] boot 2024-02-14T17:49:02.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: osdmap e130: 8 total, 8 up, 8 in 2024-02-14T17:49:02.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:49:02.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:02.068 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:01 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: osd.3 [v2:172.21.15.17:6826/2957697986,v1:172.21.15.17:6827/2957697986] boot 2024-02-14T17:49:02.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: osdmap e130: 8 total, 8 up, 8 in 2024-02-14T17:49:02.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-14T17:49:02.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: failed to encode map e130 with expected crc 2024-02-14T17:49:02.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:02.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:01 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: pgmap v179: 161 pgs: 8 peering, 20 active+undersized+degraded, 38 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-02-14T17:49:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: osdmap e131: 8 total, 8 up, 8 in 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: pgmap v179: 161 pgs: 8 peering, 20 active+undersized+degraded, 38 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: osdmap e131: 8 total, 8 up, 8 in 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: failed to encode map e131 with expected crc 2024-02-14T17:49:02.942 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:02 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: pgmap v179: 161 pgs: 8 peering, 20 active+undersized+degraded, 38 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 68/627 objects degraded (10.845%) 2024-02-14T17:49:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: osdmap e131: 8 total, 8 up, 8 in 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: failed to encode map e131 with expected crc 2024-02-14T17:49:03.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:02 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 68/627 objects degraded (10.845%), 20 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:04.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:04 smithi017 ceph-mon[83585]: pgmap v181: 161 pgs: 26 peering, 10 active+undersized+degraded, 17 active+undersized, 108 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-02-14T17:49:04.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:04 smithi017 ceph-mon[88777]: pgmap v181: 161 pgs: 26 peering, 10 active+undersized+degraded, 17 active+undersized, 108 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-02-14T17:49:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:04 smithi111 ceph-mon[68099]: pgmap v181: 161 pgs: 26 peering, 10 active+undersized+degraded, 17 active+undersized, 108 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 34/627 objects degraded (5.423%) 2024-02-14T17:49:05.439 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:05 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:49:05.234+0000 7f1ca1663700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-14T17:49:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: pgmap v182: 161 pgs: 26 peering, 135 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:06.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: pgmap v182: 161 pgs: 26 peering, 135 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: pgmap v182: 161 pgs: 26 peering, 135 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:07 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-02-14T17:49:07.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:07 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-02-14T17:49:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:07 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-02-14T17:49:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:08 smithi017 ceph-mon[83585]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:08 smithi017 ceph-mon[88777]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:08 smithi111 ceph-mon[68099]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:10 smithi017 ceph-mon[83585]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:10 smithi017 ceph-mon[88777]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:10 smithi111 ceph-mon[68099]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:49:12.144 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:49:11] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-02-14T17:49:12.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:12 smithi017 ceph-mon[88777]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-02-14T17:49:12.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:12 smithi017 ceph-mon[83585]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-02-14T17:49:12.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:12 smithi111 ceph-mon[68099]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 215 B/s rd, 0 op/s 2024-02-14T17:49:14.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:14 smithi017 ceph-mon[88777]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-02-14T17:49:14.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:14 smithi017 ceph-mon[83585]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-02-14T17:49:14.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:14 smithi111 ceph-mon[68099]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 185 B/s rd, 0 op/s 2024-02-14T17:49:16.322 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:49:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:16 smithi017 ceph-mon[88777]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-02-14T17:49:16.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:16 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:16 smithi017 ceph-mon[83585]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-02-14T17:49:16.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:16 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:16 smithi111 ceph-mon[68099]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 622 B/s rd, 0 op/s 2024-02-14T17:49:16.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:16 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:16.992 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:49:16.992 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (7m) 16s ago 28m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (6m) 4m ago 28m 84.2M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 16s ago 27m 61.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (7m) 4m ago 33m 446M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (18m) 16s ago 34m 507M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (4m) 16s ago 35m 53.6M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (5m) 4m ago 33m 36.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (4m) 16s ago 33m 39.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (7m) 16s ago 29m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (7m) 4m ago 29m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (2m) 16s ago 32m 45.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:49:16.993 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (2m) 16s ago 32m 53.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (77s) 16s ago 32m 42.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (21s) 16s ago 31m 14.3M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (31m) 4m ago 31m 55.5M 3248M 17.2.0 e1d6a67b021e abcf2fb61360 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (31m) 4m ago 31m 55.5M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (30m) 4m ago 30m 51.7M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (30m) 4m ago 30m 56.2M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (7m) 4m ago 29m 60.2M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (28m) 16s ago 28m 80.3M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:49:16.994 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (28m) 4m ago 28m 79.1M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:49:17.399 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:49:17.399 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:49:17.399 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 4 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:49:17.400 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-02-14T17:49:17.401 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 9 2024-02-14T17:49:17.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:49:17.401 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:49:17.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:17 smithi017 ceph-mon[83585]: from='client.44472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:17.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:17 smithi017 ceph-mon[83585]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:17.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:17 smithi017 ceph-mon[88777]: from='client.44472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:17.692 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:17 smithi017 ceph-mon[88777]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:17.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:17 smithi111 ceph-mon[68099]: from='client.44472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:17.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:17 smithi111 ceph-mon[68099]: from='client.44478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "4/8 daemons upgraded", 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:49:17.738 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:49:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[83585]: from='client.44484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[83585]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/4020614243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:18.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[83585]: from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:18.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[88777]: from='client.44484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:18.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[88777]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:18.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:18.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/4020614243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:18.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:18 smithi017 ceph-mon[88777]: from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:18 smithi111 ceph-mon[68099]: from='client.44484 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:18 smithi111 ceph-mon[68099]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:18 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/4020614243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:18.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:18 smithi111 ceph-mon[68099]: from='client.44496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:20 smithi017 ceph-mon[83585]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:20 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:20 smithi017 ceph-mon[88777]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:20 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:20 smithi111 ceph-mon[68099]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:20.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:20 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[88777]: Upgrade: osd.4 is safe to restart 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[83585]: Upgrade: osd.4 is safe to restart 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:21.710 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:21 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-02-14T17:49:21.710 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:21 smithi111 ceph-mon[68099]: Upgrade: osd.4 is safe to restart 2024-02-14T17:49:21.710 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:21.711 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:49:21] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-02-14T17:49:22.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:22 smithi111 ceph-mon[68099]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:22.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:22 smithi111 ceph-mon[68099]: Upgrade: Updating osd.4 2024-02-14T17:49:22.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:22 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:22.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:22 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:49:22.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:22 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:22.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:22 smithi111 ceph-mon[68099]: Deploying daemon osd.4 on smithi111 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[88777]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[88777]: Upgrade: Updating osd.4 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[88777]: Deploying daemon osd.4 on smithi111 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[83585]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[83585]: Upgrade: Updating osd.4 2024-02-14T17:49:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:23.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-14T17:49:23.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:23.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:22 smithi017 ceph-mon[83585]: Deploying daemon osd.4 on smithi111 2024-02-14T17:49:24.959 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:24 smithi111 ceph-mon[68099]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:24 smithi017 ceph-mon[83585]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:24 smithi017 ceph-mon[88777]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:25.221 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:24 smithi111 systemd[1]: Stopping Ceph osd.4 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:49:25.484 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[29025]: 2024-02-14T17:49:25.219+0000 7f9d0d00a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:49:25.484 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[29025]: 2024-02-14T17:49:25.219+0000 7f9d0d00a700 -1 osd.4 131 *** Got signal Terminated *** 2024-02-14T17:49:25.485 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[29025]: 2024-02-14T17:49:25.219+0000 7f9d0d00a700 -1 osd.4 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:49:26.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:25 smithi017 ceph-mon[83585]: osd.4 marked itself down and dead 2024-02-14T17:49:26.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:25 smithi017 ceph-mon[88777]: osd.4 marked itself down and dead 2024-02-14T17:49:26.191 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:25 smithi111 ceph-mon[68099]: osd.4 marked itself down and dead 2024-02-14T17:49:26.472 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:26 smithi111 podman[78236]: 2024-02-14 17:49:26.190206948 +0000 UTC m=+1.079226613 container died abcf2fb61360db1d6979534dd2fb56ff60073c1ed3b54787a3a14bc019255ee9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, ceph=True, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_CLEAN=True, release=754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc.) 2024-02-14T17:49:26.734 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:26 smithi111 podman[78236]: 2024-02-14 17:49:26.570207618 +0000 UTC m=+1.459227275 container cleanup abcf2fb61360db1d6979534dd2fb56ff60073c1ed3b54787a3a14bc019255ee9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.buildah.version=1.19.8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, vcs-type=git, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, name=centos-stream) 2024-02-14T17:49:26.734 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:26 smithi111 bash[78236]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4 2024-02-14T17:49:27.131 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:26 smithi111 podman[78255]: 2024-02-14 17:49:26.81329589 +0000 UTC m=+0.621729775 container remove abcf2fb61360db1d6979534dd2fb56ff60073c1ed3b54787a3a14bc019255ee9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, distribution-scope=public, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, ceph=True, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8) 2024-02-14T17:49:27.131 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:26 smithi111 ceph-mon[68099]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:27.132 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:26 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:49:27.132 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:26 smithi111 ceph-mon[68099]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.132 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:26 smithi111 ceph-mon[68099]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.132 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:26 smithi111 ceph-mon[68099]: osdmap e132: 8 total, 7 up, 8 in 2024-02-14T17:49:27.132 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:26 smithi111 ceph-mon[68099]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.132 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:26 smithi111 ceph-mon[68099]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[83585]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:49:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[83585]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[83585]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[83585]: osdmap e132: 8 total, 7 up, 8 in 2024-02-14T17:49:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[83585]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[83585]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[88777]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:27.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:49:27.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[88777]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[88777]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[88777]: osdmap e132: 8 total, 7 up, 8 in 2024-02-14T17:49:27.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[88777]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:26 smithi017 ceph-mon[88777]: failed to encode map e132 with expected crc 2024-02-14T17:49:27.434 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:27 smithi111 podman[78350]: 2024-02-14 17:49:27.130783468 +0000 UTC m=+0.108559642 container create 8373217f478aa78c6950fdaa93d47d342e1d75ccf6081ca83a1d9b716c1d2645 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240214, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:49:27.434 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:27 smithi111 podman[78350]: 2024-02-14 17:49:27.040572645 +0000 UTC m=+0.018348824 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:49:27.435 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:27 smithi111 podman[78350]: 2024-02-14 17:49:27.333831456 +0000 UTC m=+0.311607631 container init 8373217f478aa78c6950fdaa93d47d342e1d75ccf6081ca83a1d9b716c1d2645 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-02-14T17:49:27.435 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:27 smithi111 podman[78350]: 2024-02-14 17:49:27.33786338 +0000 UTC m=+0.315639566 container start 8373217f478aa78c6950fdaa93d47d342e1d75ccf6081ca83a1d9b716c1d2645 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-14T17:49:27.435 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:27 smithi111 podman[78350]: 2024-02-14 17:49:27.43379701 +0000 UTC m=+0.411573195 container attach 8373217f478aa78c6950fdaa93d47d342e1d75ccf6081ca83a1d9b716c1d2645 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-02-14T17:49:27.734 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:27 smithi111 podman[78350]: 2024-02-14 17:49:27.632511819 +0000 UTC m=+0.610287995 container died 8373217f478aa78c6950fdaa93d47d342e1d75ccf6081ca83a1d9b716c1d2645 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-14T17:49:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[88777]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[88777]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[88777]: osdmap e133: 8 total, 7 up, 8 in 2024-02-14T17:49:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[88777]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[88777]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[88777]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[83585]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[83585]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[83585]: osdmap e133: 8 total, 7 up, 8 in 2024-02-14T17:49:28.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[83585]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[83585]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:27 smithi017 ceph-mon[83585]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:27 smithi111 ceph-mon[68099]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:27 smithi111 ceph-mon[68099]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:27 smithi111 ceph-mon[68099]: osdmap e133: 8 total, 7 up, 8 in 2024-02-14T17:49:28.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:27 smithi111 ceph-mon[68099]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:27 smithi111 ceph-mon[68099]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.205 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:27 smithi111 ceph-mon[68099]: failed to encode map e133 with expected crc 2024-02-14T17:49:28.485 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 podman[78376]: 2024-02-14 17:49:28.204076095 +0000 UTC m=+0.565564123 container remove 8373217f478aa78c6950fdaa93d47d342e1d75ccf6081ca83a1d9b716c1d2645 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-14T17:49:28.485 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-14T17:49:28.485 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 systemd[1]: Stopped Ceph osd.4 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:49:28.485 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.4.service: Consumed 16.860s CPU time. 2024-02-14T17:49:28.485 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 systemd[1]: Starting Ceph osd.4 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:49:28.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:28 smithi111 ceph-mon[68099]: pgmap v195: 161 pgs: 13 active+undersized+degraded, 9 peering, 12 stale+active+clean, 19 active+undersized, 108 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-02-14T17:49:28.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:28.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:28 smithi111 ceph-mon[68099]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-14T17:49:28.925 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:28 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 57/627 objects degraded (9.091%), 13 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:28.926 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 podman[78480]: 2024-02-14 17:49:28.638281684 +0000 UTC m=+0.105886845 container create 1bddaa784c5a15b30dd15f451886436b4cb7d05286b1d3471da28b2f34de7e98 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:49:28.926 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 podman[78480]: 2024-02-14 17:49:28.550167067 +0000 UTC m=+0.017772235 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:49:28.926 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 podman[78480]: 2024-02-14 17:49:28.858436331 +0000 UTC m=+0.326041502 container init 1bddaa784c5a15b30dd15f451886436b4cb7d05286b1d3471da28b2f34de7e98 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:49:28.926 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 podman[78480]: 2024-02-14 17:49:28.861879565 +0000 UTC m=+0.329484743 container start 1bddaa784c5a15b30dd15f451886436b4cb7d05286b1d3471da28b2f34de7e98 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-02-14T17:49:28.926 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:28 smithi111 podman[78480]: 2024-02-14 17:49:28.924583815 +0000 UTC m=+0.392188987 container attach 1bddaa784c5a15b30dd15f451886436b4cb7d05286b1d3471da28b2f34de7e98 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:49:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[83585]: pgmap v195: 161 pgs: 13 active+undersized+degraded, 9 peering, 12 stale+active+clean, 19 active+undersized, 108 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-02-14T17:49:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[83585]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-14T17:49:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 57/627 objects degraded (9.091%), 13 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:29.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[88777]: pgmap v195: 161 pgs: 13 active+undersized+degraded, 9 peering, 12 stale+active+clean, 19 active+undersized, 108 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 57/627 objects degraded (9.091%) 2024-02-14T17:49:29.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:29.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[88777]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-02-14T17:49:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:28 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 57/627 objects degraded (9.091%), 13 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate[78495]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate[78495]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 bash[78480]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 bash[78480]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 bash[78480]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 bash[78480]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 bash[78480]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 bash[78480]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 bash[78480]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate[78495]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate[78495]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-02-14T17:49:29.629 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate[78495]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-02-14T17:49:29.630 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate[78495]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-02-14T17:49:29.630 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate[78495]: --> ceph-volume raw activate successful for osd ID: 4 2024-02-14T17:49:29.630 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 podman[78480]: 2024-02-14 17:49:29.323593937 +0000 UTC m=+0.791199090 container died 1bddaa784c5a15b30dd15f451886436b4cb7d05286b1d3471da28b2f34de7e98 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-14T17:49:30.234 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:29 smithi111 podman[78611]: 2024-02-14 17:49:29.903279477 +0000 UTC m=+0.574660163 container remove 1bddaa784c5a15b30dd15f451886436b4cb7d05286b1d3471da28b2f34de7e98 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-activate, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-14T17:49:30.234 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 podman[78678]: 2024-02-14 17:49:30.019755475 +0000 UTC m=+0.018437902 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:49:30.234 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 podman[78678]: 2024-02-14 17:49:30.137474499 +0000 UTC m=+0.136156888 container create c579b5723b701a2e9ad4e6ff6d29c36618e7715d2661821834e20d0d171a2b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-14T17:49:30.734 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 podman[78678]: 2024-02-14 17:49:30.33147418 +0000 UTC m=+0.330156568 container init c579b5723b701a2e9ad4e6ff6d29c36618e7715d2661821834e20d0d171a2b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:49:30.734 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 podman[78678]: 2024-02-14 17:49:30.334691162 +0000 UTC m=+0.333373551 container start c579b5723b701a2e9ad4e6ff6d29c36618e7715d2661821834e20d0d171a2b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:49:30.735 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 ceph-osd[78697]: -- 172.21.15.111:0/1339130365 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559a8e2051e0 con 0x559a8e1d7400 2024-02-14T17:49:30.735 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 bash[78678]: c579b5723b701a2e9ad4e6ff6d29c36618e7715d2661821834e20d0d171a2b45 2024-02-14T17:49:30.735 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 systemd[1]: Started Ceph osd.4 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[83585]: pgmap v196: 161 pgs: 23 active+undersized+degraded, 9 peering, 6 stale+active+clean, 26 active+undersized, 97 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 90/627 objects degraded (14.354%) 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[88777]: pgmap v196: 161 pgs: 23 active+undersized+degraded, 9 peering, 6 stale+active+clean, 26 active+undersized, 97 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 90/627 objects degraded (14.354%) 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:31.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:30 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:31.234 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:49:30.873+0000 7f52592257c0 -1 Falling back to public interface 2024-02-14T17:49:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:30 smithi111 ceph-mon[68099]: pgmap v196: 161 pgs: 23 active+undersized+degraded, 9 peering, 6 stale+active+clean, 26 active+undersized, 97 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 90/627 objects degraded (14.354%) 2024-02-14T17:49:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:30 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:31.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:30 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:31.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:30 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:32.091 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:49:31] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-02-14T17:49:33.000 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:32 smithi111 ceph-mon[68099]: pgmap v197: 161 pgs: 25 active+undersized+degraded, 9 peering, 34 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 95/627 objects degraded (15.152%) 2024-02-14T17:49:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:32 smithi017 ceph-mon[83585]: pgmap v197: 161 pgs: 25 active+undersized+degraded, 9 peering, 34 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 95/627 objects degraded (15.152%) 2024-02-14T17:49:33.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:32 smithi017 ceph-mon[88777]: pgmap v197: 161 pgs: 25 active+undersized+degraded, 9 peering, 34 active+undersized, 93 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 95/627 objects degraded (15.152%) 2024-02-14T17:49:33.720 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:49:33.592+0000 7f52592257c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-02-14T17:49:33.721 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:49:33.624+0000 7f52592257c0 -1 osd.4 131 log_to_monitors true 2024-02-14T17:49:33.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:33 smithi111 ceph-mon[68099]: from='osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:49:33.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:33 smithi111 ceph-mon[68099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:49:33.984 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:49:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:49:33.834+0000 7f524a67d700 -1 osd.4 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:49:34.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:33 smithi017 ceph-mon[83585]: from='osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:49:34.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:33 smithi017 ceph-mon[83585]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:49:34.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:33 smithi017 ceph-mon[88777]: from='osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:49:34.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:33 smithi017 ceph-mon[88777]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: pgmap v198: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: failed to encode map e134 with expected crc 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: failed to encode map e134 with expected crc 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: from='osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: osdmap e134: 8 total, 7 up, 8 in 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: pgmap v198: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: failed to encode map e134 with expected crc 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: failed to encode map e134 with expected crc 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: from='osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: osdmap e134: 8 total, 7 up, 8 in 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:35.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:35.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:34 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: pgmap v198: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: failed to encode map e134 with expected crc 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: failed to encode map e134 with expected crc 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: from='osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: osdmap e134: 8 total, 7 up, 8 in 2024-02-14T17:49:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:35.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:35.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:34 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690] boot 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: osdmap e135: 8 total, 8 up, 8 in 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690] boot 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: osdmap e135: 8 total, 8 up, 8 in 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:35 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: osd.4 [v2:172.21.15.111:6800/705454690,v1:172.21.15.111:6801/705454690] boot 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: osdmap e135: 8 total, 8 up, 8 in 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: failed to encode map e135 with expected crc 2024-02-14T17:49:36.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:35 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-14T17:49:36.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: pgmap v201: 161 pgs: 4 peering, 26 active+undersized+degraded, 38 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 102/627 objects degraded (16.268%) 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: osdmap e136: 8 total, 8 up, 8 in 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: failed to encode map e136 with expected crc 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: failed to encode map e136 with expected crc 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: failed to encode map e136 with expected crc 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: failed to encode map e136 with expected crc 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: failed to encode map e136 with expected crc 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: failed to encode map e136 with expected crc 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:36.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: pgmap v201: 161 pgs: 4 peering, 26 active+undersized+degraded, 38 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 102/627 objects degraded (16.268%) 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: osdmap e136: 8 total, 8 up, 8 in 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:37.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: pgmap v201: 161 pgs: 4 peering, 26 active+undersized+degraded, 38 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 102/627 objects degraded (16.268%) 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: osdmap e136: 8 total, 8 up, 8 in 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: failed to encode map e136 with expected crc 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:37.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:37.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:39.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:38 smithi111 ceph-mon[68099]: pgmap v203: 161 pgs: 4 peering, 15 active+undersized+degraded, 19 active+undersized, 123 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/627 objects degraded (7.177%) 2024-02-14T17:49:39.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:39.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:38 smithi017 ceph-mon[83585]: pgmap v203: 161 pgs: 4 peering, 15 active+undersized+degraded, 19 active+undersized, 123 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/627 objects degraded (7.177%) 2024-02-14T17:49:39.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:39.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:38 smithi017 ceph-mon[88777]: pgmap v203: 161 pgs: 4 peering, 15 active+undersized+degraded, 19 active+undersized, 123 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/627 objects degraded (7.177%) 2024-02-14T17:49:39.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:40.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: pgmap v204: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:40.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: pgmap v205: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: pgmap v206: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: Upgrade: osd.5 is safe to restart 2024-02-14T17:49:40.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/627 objects degraded (7.177%), 15 pgs degraded) 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: pgmap v204: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: pgmap v205: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: pgmap v206: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-14T17:49:40.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-14T17:49:40.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: Upgrade: osd.5 is safe to restart 2024-02-14T17:49:40.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:40 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/627 objects degraded (7.177%), 15 pgs degraded) 2024-02-14T17:49:40.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: pgmap v204: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: pgmap v205: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: pgmap v206: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:40.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-14T17:49:40.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-02-14T17:49:40.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: Upgrade: osd.5 is safe to restart 2024-02-14T17:49:40.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:40 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/627 objects degraded (7.177%), 15 pgs degraded) 2024-02-14T17:49:41.965 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:41 smithi111 ceph-mon[68099]: Upgrade: Updating osd.5 2024-02-14T17:49:41.965 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:41 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:41.965 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:41 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:49:41.966 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:41 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:41.966 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:41 smithi111 ceph-mon[68099]: Deploying daemon osd.5 on smithi111 2024-02-14T17:49:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:49:41] "GET /metrics HTTP/1.1" 200 38060 "" "Prometheus/2.43.0" 2024-02-14T17:49:42.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[83585]: Upgrade: Updating osd.5 2024-02-14T17:49:42.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:42.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:49:42.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:42.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[83585]: Deploying daemon osd.5 on smithi111 2024-02-14T17:49:42.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[88777]: Upgrade: Updating osd.5 2024-02-14T17:49:42.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:42.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-14T17:49:42.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:42.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:41 smithi017 ceph-mon[88777]: Deploying daemon osd.5 on smithi111 2024-02-14T17:49:42.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:42 smithi111 ceph-mon[68099]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-02-14T17:49:43.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:42 smithi017 ceph-mon[83585]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-02-14T17:49:43.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:42 smithi017 ceph-mon[88777]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-02-14T17:49:44.734 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:44 smithi111 systemd[1]: Stopping Ceph osd.5 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:49:44.735 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[31912]: 2024-02-14T17:49:44.621+0000 7f4918be6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:49:44.735 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[31912]: 2024-02-14T17:49:44.621+0000 7f4918be6700 -1 osd.5 136 *** Got signal Terminated *** 2024-02-14T17:49:44.735 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[31912]: 2024-02-14T17:49:44.621+0000 7f4918be6700 -1 osd.5 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:49:45.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:44 smithi017 ceph-mon[88777]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-02-14T17:49:45.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:44 smithi017 ceph-mon[88777]: osd.5 marked itself down and dead 2024-02-14T17:49:45.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:44 smithi017 ceph-mon[83585]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-02-14T17:49:45.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:44 smithi017 ceph-mon[83585]: osd.5 marked itself down and dead 2024-02-14T17:49:45.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:44 smithi111 ceph-mon[68099]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-02-14T17:49:45.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:44 smithi111 ceph-mon[68099]: osd.5 marked itself down and dead 2024-02-14T17:49:45.699 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:45 smithi111 podman[83191]: 2024-02-14 17:49:45.409330455 +0000 UTC m=+0.887820564 container died 20ea7a4da4fae9f84b603724cc03f9218d267b91b454de4f69381ea9d4286708 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, name=centos-stream, ceph=True, architecture=x86_64, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-02-14T17:49:45.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: osdmap e137: 8 total, 7 up, 8 in 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: failed to encode map e137 with expected crc 2024-02-14T17:49:45.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: pgmap v210: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:49:45.986 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:45 smithi111 ceph-mon[68099]: osdmap e138: 8 total, 7 up, 8 in 2024-02-14T17:49:45.986 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:45 smithi111 podman[83191]: 2024-02-14 17:49:45.897874137 +0000 UTC m=+1.376364266 container cleanup 20ea7a4da4fae9f84b603724cc03f9218d267b91b454de4f69381ea9d4286708 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.expose-services=, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-14T17:49:45.986 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:45 smithi111 bash[83191]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5 2024-02-14T17:49:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:49:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: osdmap e137: 8 total, 7 up, 8 in 2024-02-14T17:49:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: pgmap v210: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:49:46.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[83585]: osdmap e138: 8 total, 7 up, 8 in 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: osdmap e137: 8 total, 7 up, 8 in 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: failed to encode map e137 with expected crc 2024-02-14T17:49:46.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: pgmap v210: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-14T17:49:46.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:45 smithi017 ceph-mon[88777]: osdmap e138: 8 total, 7 up, 8 in 2024-02-14T17:49:46.464 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:46 smithi111 podman[83209]: 2024-02-14 17:49:46.148683636 +0000 UTC m=+0.740923388 container remove 20ea7a4da4fae9f84b603724cc03f9218d267b91b454de4f69381ea9d4286708 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, name=centos-stream, RELEASE=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, release=754) 2024-02-14T17:49:46.734 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:46 smithi111 podman[83301]: 2024-02-14 17:49:46.366745592 +0000 UTC m=+0.018263438 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:49:46.734 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:46 smithi111 podman[83301]: 2024-02-14 17:49:46.474874884 +0000 UTC m=+0.126392776 container create b8bfbe52ef8f01831bbb88185feb0bca6d48535dc56496968cbd59014698d679 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214) 2024-02-14T17:49:46.734 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:46 smithi111 podman[83301]: 2024-02-14 17:49:46.695010956 +0000 UTC m=+0.346528808 container init b8bfbe52ef8f01831bbb88185feb0bca6d48535dc56496968cbd59014698d679 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True) 2024-02-14T17:49:46.735 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:46 smithi111 podman[83301]: 2024-02-14 17:49:46.698119079 +0000 UTC m=+0.349636926 container start b8bfbe52ef8f01831bbb88185feb0bca6d48535dc56496968cbd59014698d679 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-14T17:49:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[83585]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:46 smithi017 ceph-mon[88777]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:46 smithi111 podman[83301]: 2024-02-14 17:49:46.777431705 +0000 UTC m=+0.428949566 container attach b8bfbe52ef8f01831bbb88185feb0bca6d48535dc56496968cbd59014698d679 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:49:47.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:46 smithi111 podman[83301]: 2024-02-14 17:49:46.984377526 +0000 UTC m=+0.635895374 container died b8bfbe52ef8f01831bbb88185feb0bca6d48535dc56496968cbd59014698d679 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-14T17:49:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:46 smithi111 ceph-mon[68099]: failed to encode map e138 with expected crc 2024-02-14T17:49:47.905 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:47 smithi111 podman[83327]: 2024-02-14 17:49:47.614378994 +0000 UTC m=+0.624734852 container remove b8bfbe52ef8f01831bbb88185feb0bca6d48535dc56496968cbd59014698d679 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240214) 2024-02-14T17:49:47.905 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:47 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.5.service: Deactivated successfully. 2024-02-14T17:49:47.905 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:47 smithi111 systemd[1]: Stopped Ceph osd.5 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:49:47.905 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:47 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.5.service: Consumed 16.014s CPU time. 2024-02-14T17:49:47.906 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:47 smithi111 systemd[1]: Starting Ceph osd.5 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:49:48.108 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:49:48.170 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:47 smithi111 ceph-mon[68099]: pgmap v212: 161 pgs: 10 active+undersized+degraded, 11 stale+active+clean, 17 active+undersized, 123 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:49:48.170 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:47 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:48.171 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 podman[83428]: 2024-02-14 17:49:47.958028687 +0000 UTC m=+0.017877262 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:49:48.171 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 podman[83428]: 2024-02-14 17:49:48.057006475 +0000 UTC m=+0.116855079 container create d17482b3d4ecdfbd815a0c8e61d7d2c8ee9fc845f687c3d39115be3dc19a4870 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate, org.label-schema.build-date=20240214, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:49:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:47 smithi017 ceph-mon[83585]: pgmap v212: 161 pgs: 10 active+undersized+degraded, 11 stale+active+clean, 17 active+undersized, 123 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:49:48.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:47 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:48.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:47 smithi017 ceph-mon[88777]: pgmap v212: 161 pgs: 10 active+undersized+degraded, 11 stale+active+clean, 17 active+undersized, 123 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 30/627 objects degraded (4.785%) 2024-02-14T17:49:48.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:47 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:48.484 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 podman[83428]: 2024-02-14 17:49:48.268514139 +0000 UTC m=+0.328362700 container init d17482b3d4ecdfbd815a0c8e61d7d2c8ee9fc845f687c3d39115be3dc19a4870 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-14T17:49:48.484 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 podman[83428]: 2024-02-14 17:49:48.272129884 +0000 UTC m=+0.331978459 container start d17482b3d4ecdfbd815a0c8e61d7d2c8ee9fc845f687c3d39115be3dc19a4870 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-14T17:49:48.484 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 podman[83428]: 2024-02-14 17:49:48.326303856 +0000 UTC m=+0.386152430 container attach d17482b3d4ecdfbd815a0c8e61d7d2c8ee9fc845f687c3d39115be3dc19a4870 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:49:48.773 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (7m) 48s ago 29m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (7m) 14s ago 29m 89.9M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 48s ago 28m 61.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (8m) 14s ago 33m 452M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (18m) 48s ago 35m 507M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (5m) 48s ago 35m 53.6M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (6m) 14s ago 34m 44.5M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (4m) 48s ago 34m 39.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (8m) 48s ago 29m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (8m) 14s ago 29m 16.8M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (3m) 48s ago 33m 45.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:49:48.774 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (2m) 48s ago 33m 53.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (109s) 48s ago 32m 42.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (53s) 48s ago 32m 14.3M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (18s) 14s ago 32m 15.1M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (31m) 14s ago 31m 60.7M 3248M 17.2.0 e1d6a67b021e 20ea7a4da4fa 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (31m) 14s ago 31m 56.2M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (30m) 14s ago 30m 62.6M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (7m) 14s ago 30m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (28m) 48s ago 28m 80.3M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:49:48.775 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (28m) 14s ago 28m 80.7M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:49:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 30/627 objects degraded (4.785%), 10 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-mon[68099]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-mon[68099]: from='client.44514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-mon[68099]: from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:48.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate[83442]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-14T17:49:48.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate[83442]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 bash[83428]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 bash[83428]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 bash[83428]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 bash[83428]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 bash[83428]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 bash[83428]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 bash[83428]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate[83442]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate[83442]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate[83442]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate[83442]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate[83442]: --> ceph-volume raw activate successful for osd ID: 5 2024-02-14T17:49:48.985 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:48 smithi111 podman[83428]: 2024-02-14 17:49:48.733399186 +0000 UTC m=+0.793247748 container died d17482b3d4ecdfbd815a0c8e61d7d2c8ee9fc845f687c3d39115be3dc19a4870 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:49:49.182 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 5 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 10 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:49:49.183 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:49:49.183 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 30/627 objects degraded (4.785%), 10 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:49.183 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[83585]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:49.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[83585]: from='client.44514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:49.184 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[83585]: from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:49.184 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 30/627 objects degraded (4.785%), 10 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:49.184 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[88777]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:49.184 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[88777]: from='client.44514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:49.184 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:48 smithi017 ceph-mon[88777]: from='client.44520 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:49.521 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:49:49.521 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:49:49.522 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:49:49.522 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-14T17:49:49.522 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:49:49.522 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "5/8 daemons upgraded", 2024-02-14T17:49:49.522 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:49:49.522 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:49:49.522 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:49:49.564 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 podman[83575]: 2024-02-14 17:49:49.313252102 +0000 UTC m=+0.574747530 container remove d17482b3d4ecdfbd815a0c8e61d7d2c8ee9fc845f687c3d39115be3dc19a4870 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:49:49.565 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 podman[83642]: 2024-02-14 17:49:49.429954805 +0000 UTC m=+0.019081280 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:49:49.816 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 podman[83642]: 2024-02-14 17:49:49.564148888 +0000 UTC m=+0.153275344 container create 99350ba8e2aa6b7369ae09c94421d252999ac4357833ff2b76563b2e10230623 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:49:49.816 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 podman[83642]: 2024-02-14 17:49:49.791759661 +0000 UTC m=+0.380886120 container init 99350ba8e2aa6b7369ae09c94421d252999ac4357833ff2b76563b2e10230623 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-14T17:49:49.816 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 podman[83642]: 2024-02-14 17:49:49.795089002 +0000 UTC m=+0.384215465 container start 99350ba8e2aa6b7369ae09c94421d252999ac4357833ff2b76563b2e10230623 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-14T17:49:50.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:49 smithi111 ceph-mon[68099]: pgmap v213: 161 pgs: 19 active+undersized+degraded, 5 stale+active+clean, 30 active+undersized, 107 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-14T17:49:50.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:49 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/753163300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:50.080 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:49 smithi111 ceph-mon[68099]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:50.081 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 ceph-osd[83662]: -- 172.21.15.111:0/1070183468 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55efb4db91e0 con 0x55efb4d8ac00 2024-02-14T17:49:50.081 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 bash[83642]: 99350ba8e2aa6b7369ae09c94421d252999ac4357833ff2b76563b2e10230623 2024-02-14T17:49:50.081 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:49 smithi111 systemd[1]: Started Ceph osd.5 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:49:50.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:49 smithi017 ceph-mon[83585]: pgmap v213: 161 pgs: 19 active+undersized+degraded, 5 stale+active+clean, 30 active+undersized, 107 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-14T17:49:50.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:49 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/753163300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:50.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:49 smithi017 ceph-mon[83585]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:49 smithi017 ceph-mon[88777]: pgmap v213: 161 pgs: 19 active+undersized+degraded, 5 stale+active+clean, 30 active+undersized, 107 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 56/627 objects degraded (8.931%) 2024-02-14T17:49:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:49 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/753163300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:49 smithi017 ceph-mon[88777]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:49:50.734 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:49:50.333+0000 7f94551107c0 -1 Falling back to public interface 2024-02-14T17:49:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:50 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:50 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:51.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:50 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:50 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:50 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:51.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:50 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:50 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:50 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:51.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:50 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:52.030 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:49:51] "GET /metrics HTTP/1.1" 200 38199 "" "Prometheus/2.43.0" 2024-02-14T17:49:52.326 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:52.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:52.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:52.327 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:52 smithi111 ceph-mon[68099]: pgmap v214: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-14T17:49:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:52.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[88777]: pgmap v214: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-14T17:49:52.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:52.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:49:52.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:52.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:52 smithi017 ceph-mon[83585]: pgmap v214: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-14T17:49:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:53 smithi017 ceph-mon[83585]: from='osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:49:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:53 smithi017 ceph-mon[83585]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:49:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:53 smithi017 ceph-mon[88777]: from='osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:49:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:53 smithi017 ceph-mon[88777]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:49:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:53 smithi111 ceph-mon[68099]: from='osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:49:53.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:53 smithi111 ceph-mon[68099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-14T17:49:53.484 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:49:53.053+0000 7f94551107c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-02-14T17:49:53.484 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:49:53.081+0000 7f94551107c0 -1 osd.5 136 log_to_monitors true 2024-02-14T17:49:53.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:49:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:49:53.593+0000 7f9442560700 -1 osd.5 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:49:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[83585]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-14T17:49:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[83585]: osdmap e139: 8 total, 7 up, 8 in 2024-02-14T17:49:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[83585]: from='osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[83585]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:54.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[83585]: pgmap v216: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-14T17:49:54.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[83585]: from='osd.5 ' entity='osd.5' 2024-02-14T17:49:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[88777]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-14T17:49:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[88777]: osdmap e139: 8 total, 7 up, 8 in 2024-02-14T17:49:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[88777]: from='osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[88777]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[88777]: pgmap v216: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-14T17:49:54.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:54 smithi017 ceph-mon[88777]: from='osd.5 ' entity='osd.5' 2024-02-14T17:49:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:54 smithi111 ceph-mon[68099]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-14T17:49:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:54 smithi111 ceph-mon[68099]: osdmap e139: 8 total, 7 up, 8 in 2024-02-14T17:49:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:54 smithi111 ceph-mon[68099]: from='osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:54 smithi111 ceph-mon[68099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:49:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:54 smithi111 ceph-mon[68099]: pgmap v216: 161 pgs: 24 active+undersized+degraded, 35 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 75/627 objects degraded (11.962%) 2024-02-14T17:49:54.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:54 smithi111 ceph-mon[68099]: from='osd.5 ' entity='osd.5' 2024-02-14T17:49:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:55.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760] boot 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: osdmap e140: 8 total, 8 up, 8 in 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760] boot 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: osdmap e140: 8 total, 8 up, 8 in 2024-02-14T17:49:55.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:55 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:49:55.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: failed to encode map e139 with expected crc 2024-02-14T17:49:55.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:55.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:55.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:49:55.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: osd.5 [v2:172.21.15.111:6808/2985716760,v1:172.21.15.111:6809/2985716760] boot 2024-02-14T17:49:55.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: osdmap e140: 8 total, 8 up, 8 in 2024-02-14T17:49:55.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:55 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-14T17:49:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: pgmap v218: 161 pgs: 5 peering, 22 active+undersized+degraded, 32 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:49:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:56.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:56.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 69/627 objects degraded (11.005%), 22 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:56.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:56 smithi111 ceph-mon[68099]: osdmap e141: 8 total, 8 up, 8 in 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: pgmap v218: 161 pgs: 5 peering, 22 active+undersized+degraded, 32 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 69/627 objects degraded (11.005%), 22 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[83585]: osdmap e141: 8 total, 8 up, 8 in 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: failed to encode map e140 with expected crc 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: pgmap v218: 161 pgs: 5 peering, 22 active+undersized+degraded, 32 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 69/627 objects degraded (11.005%) 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 69/627 objects degraded (11.005%), 22 pgs degraded (PG_DEGRADED) 2024-02-14T17:49:56.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:56 smithi017 ceph-mon[88777]: osdmap e141: 8 total, 8 up, 8 in 2024-02-14T17:49:57.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:57 smithi111 ceph-mon[68099]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:57 smithi111 ceph-mon[68099]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:57 smithi111 ceph-mon[68099]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:57 smithi111 ceph-mon[68099]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:57 smithi111 ceph-mon[68099]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[83585]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[83585]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[83585]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[83585]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[83585]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[88777]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[88777]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[88777]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[88777]: failed to encode map e141 with expected crc 2024-02-14T17:49:57.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:57 smithi017 ceph-mon[88777]: failed to encode map e141 with expected crc 2024-02-14T17:49:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:58 smithi111 ceph-mon[68099]: pgmap v220: 161 pgs: 5 peering, 16 active+undersized+degraded, 21 active+undersized, 119 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 46/627 objects degraded (7.337%) 2024-02-14T17:49:58.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:58 smithi017 ceph-mon[83585]: pgmap v220: 161 pgs: 5 peering, 16 active+undersized+degraded, 21 active+undersized, 119 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 46/627 objects degraded (7.337%) 2024-02-14T17:49:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:58 smithi017 ceph-mon[88777]: pgmap v220: 161 pgs: 5 peering, 16 active+undersized+degraded, 21 active+undersized, 119 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 46/627 objects degraded (7.337%) 2024-02-14T17:49:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:49:59.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:49:58 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:49:58.775+0000 7f1ca1663700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-02-14T17:49:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: pgmap v221: 161 pgs: 5 peering, 2 active+undersized+degraded, 4 active+undersized, 150 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-02-14T17:49:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: pgmap v222: 161 pgs: 5 peering, 2 active+undersized+degraded, 4 active+undersized, 150 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-02-14T17:49:59.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[88777]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:59.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: pgmap v221: 161 pgs: 5 peering, 2 active+undersized+degraded, 4 active+undersized, 150 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: pgmap v222: 161 pgs: 5 peering, 2 active+undersized+degraded, 4 active+undersized, 150 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:49:59.942 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:49:59 smithi017 ceph-mon[83585]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-02-14T17:49:59.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:49:59.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:49:59.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: pgmap v221: 161 pgs: 5 peering, 2 active+undersized+degraded, 4 active+undersized, 150 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: pgmap v222: 161 pgs: 5 peering, 2 active+undersized+degraded, 4 active+undersized, 150 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s; 2/627 objects degraded (0.319%) 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:49:59.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:49:59 smithi111 ceph-mon[68099]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-02-14T17:50:00.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[83585]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded 2024-02-14T17:50:00.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[83585]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[83585]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[83585]: [WRN] PG_DEGRADED: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[83585]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[83585]: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[88777]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[88777]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[88777]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[88777]: [WRN] PG_DEGRADED: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[88777]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-02-14T17:50:00.941 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:00 smithi017 ceph-mon[88777]: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-02-14T17:50:00.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:00 smithi111 ceph-mon[68099]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded 2024-02-14T17:50:00.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:00 smithi111 ceph-mon[68099]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:50:00.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:00 smithi111 ceph-mon[68099]: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:50:00.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:00 smithi111 ceph-mon[68099]: [WRN] PG_DEGRADED: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded 2024-02-14T17:50:00.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:00 smithi111 ceph-mon[68099]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-02-14T17:50:00.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:00 smithi111 ceph-mon[68099]: pg 6.1a is active+undersized+degraded, acting [4,1] 2024-02-14T17:50:01.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:01 smithi017 ceph-mon[83585]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-14T17:50:01.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:01 smithi017 ceph-mon[88777]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-14T17:50:01.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:50:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:50:01] "GET /metrics HTTP/1.1" 200 38199 "" "Prometheus/2.43.0" 2024-02-14T17:50:01.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:01 smithi111 ceph-mon[68099]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-14T17:50:02.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:02 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded) 2024-02-14T17:50:02.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:02 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded) 2024-02-14T17:50:02.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:02 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/627 objects degraded (0.319%), 2 pgs degraded) 2024-02-14T17:50:03.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:03 smithi017 ceph-mon[83585]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 684 B/s rd, 0 op/s 2024-02-14T17:50:03.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:03 smithi017 ceph-mon[88777]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 684 B/s rd, 0 op/s 2024-02-14T17:50:03.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:03 smithi111 ceph-mon[68099]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 684 B/s rd, 0 op/s 2024-02-14T17:50:05.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:05 smithi017 ceph-mon[83585]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-02-14T17:50:05.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:05 smithi017 ceph-mon[88777]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-02-14T17:50:05.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:05 smithi111 ceph-mon[68099]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-02-14T17:50:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[83585]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:50:07.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:07.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:07.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:07.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:07 smithi017 ceph-mon[88777]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:50:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:07 smithi111 ceph-mon[68099]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:50:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:09 smithi017 ceph-mon[83585]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:50:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:09 smithi017 ceph-mon[88777]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:50:09.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:09 smithi111 ceph-mon[68099]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:50:11.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:11 smithi017 ceph-mon[83585]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:11.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:11 smithi017 ceph-mon[88777]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:11.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:50:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:50:11] "GET /metrics HTTP/1.1" 200 38209 "" "Prometheus/2.43.0" 2024-02-14T17:50:11.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:11 smithi111 ceph-mon[68099]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:13.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:13 smithi017 ceph-mon[83585]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:13.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:13 smithi017 ceph-mon[88777]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:13.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:13 smithi111 ceph-mon[68099]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:14.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:14 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:50:14.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:14 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:50:14.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:14 smithi111 ceph-mon[68099]: Upgrade: osd.6 is safe to restart 2024-02-14T17:50:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:14 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:50:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:14 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:50:15.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:14 smithi017 ceph-mon[88777]: Upgrade: osd.6 is safe to restart 2024-02-14T17:50:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:14 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:50:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:14 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-02-14T17:50:15.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:14 smithi017 ceph-mon[83585]: Upgrade: osd.6 is safe to restart 2024-02-14T17:50:15.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:15 smithi111 ceph-mon[68099]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:15.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:15 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:15.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:15 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:50:15.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:15 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[83585]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:50:16.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:16.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[88777]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:16.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:16.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-14T17:50:16.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:15 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:16.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:16 smithi111 ceph-mon[68099]: Upgrade: Updating osd.6 2024-02-14T17:50:16.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:16 smithi111 ceph-mon[68099]: Deploying daemon osd.6 on smithi111 2024-02-14T17:50:17.062 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:16 smithi017 ceph-mon[88777]: Upgrade: Updating osd.6 2024-02-14T17:50:17.062 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:16 smithi017 ceph-mon[88777]: Deploying daemon osd.6 on smithi111 2024-02-14T17:50:17.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:16 smithi017 ceph-mon[83585]: Upgrade: Updating osd.6 2024-02-14T17:50:17.063 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:16 smithi017 ceph-mon[83585]: Deploying daemon osd.6 on smithi111 2024-02-14T17:50:17.956 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:17 smithi111 ceph-mon[68099]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:18.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:17 smithi017 ceph-mon[83585]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:18.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:17 smithi017 ceph-mon[88777]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:18.536 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:18 smithi111 systemd[1]: Stopping Ceph osd.6 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:50:18.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:18.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:18 smithi111 ceph-mon[68099]: osd.6 marked itself down and dead 2024-02-14T17:50:18.984 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[34822]: 2024-02-14T17:50:18.534+0000 7f877331b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:50:18.985 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[34822]: 2024-02-14T17:50:18.534+0000 7f877331b700 -1 osd.6 141 *** Got signal Terminated *** 2024-02-14T17:50:18.985 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:18 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[34822]: 2024-02-14T17:50:18.534+0000 7f877331b700 -1 osd.6 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:50:19.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:19.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:18 smithi017 ceph-mon[83585]: osd.6 marked itself down and dead 2024-02-14T17:50:19.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:19.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:18 smithi017 ceph-mon[88777]: osd.6 marked itself down and dead 2024-02-14T17:50:19.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:19 smithi111 podman[87735]: 2024-02-14 17:50:19.230074584 +0000 UTC m=+0.816175906 container died 9780cee91e7698fad11f72171abcf72299cd43f6a24cc48dd6d75de8ce20b50c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, RELEASE=HEAD, GIT_BRANCH=HEAD, name=centos-stream, architecture=x86_64, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-02-14T17:50:19.903 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:50:19.904 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:19 smithi111 ceph-mon[68099]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:19.904 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:19 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:50:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:19 smithi111 ceph-mon[68099]: failed to encode map e142 with expected crc 2024-02-14T17:50:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:19 smithi111 ceph-mon[68099]: osdmap e142: 8 total, 7 up, 8 in 2024-02-14T17:50:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:19 smithi111 ceph-mon[68099]: failed to encode map e142 with expected crc 2024-02-14T17:50:19.905 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:19 smithi111 ceph-mon[68099]: failed to encode map e142 with expected crc 2024-02-14T17:50:19.905 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:19 smithi111 podman[87735]: 2024-02-14 17:50:19.626693067 +0000 UTC m=+1.212794373 container cleanup 9780cee91e7698fad11f72171abcf72299cd43f6a24cc48dd6d75de8ce20b50c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.openshift.expose-services=, vendor=Red Hat, Inc., ceph=True) 2024-02-14T17:50:19.905 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:19 smithi111 bash[87735]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[83585]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[83585]: failed to encode map e142 with expected crc 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[83585]: osdmap e142: 8 total, 7 up, 8 in 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[83585]: failed to encode map e142 with expected crc 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[83585]: failed to encode map e142 with expected crc 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[88777]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[88777]: failed to encode map e142 with expected crc 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[88777]: osdmap e142: 8 total, 7 up, 8 in 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[88777]: failed to encode map e142 with expected crc 2024-02-14T17:50:20.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:19 smithi017 ceph-mon[88777]: failed to encode map e142 with expected crc 2024-02-14T17:50:20.234 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:19 smithi111 podman[87752]: 2024-02-14 17:50:19.903606607 +0000 UTC m=+0.677414598 container remove 9780cee91e7698fad11f72171abcf72299cd43f6a24cc48dd6d75de8ce20b50c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870) 2024-02-14T17:50:20.520 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:20 smithi111 podman[87856]: 2024-02-14 17:50:20.169999046 +0000 UTC m=+0.021098909 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:50:20.520 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:20 smithi111 podman[87856]: 2024-02-14 17:50:20.279774629 +0000 UTC m=+0.130874492 container create d7374f15561bc4a975c322163fd6238249e81d14811d99b6012b348ec7a97a5c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-02-14T17:50:20.520 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:20 smithi111 podman[87856]: 2024-02-14 17:50:20.516153461 +0000 UTC m=+0.367253331 container init d7374f15561bc4a975c322163fd6238249e81d14811d99b6012b348ec7a97a5c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:50:20.521 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:20 smithi111 podman[87856]: 2024-02-14 17:50:20.519832052 +0000 UTC m=+0.370931920 container start d7374f15561bc4a975c322163fd6238249e81d14811d99b6012b348ec7a97a5c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:50:20.570 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:50:20.570 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (8m) 80s ago 29m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:50:20.570 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (7m) 26s ago 29m 87.1M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:50:20.570 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 80s ago 29m 61.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:50:20.570 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (9m) 26s ago 34m 452M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:50:20.570 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (19m) 80s ago 35m 507M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (5m) 80s ago 36m 53.6M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (6m) 26s ago 34m 45.8M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (5m) 80s ago 34m 39.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (8m) 80s ago 30m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (8m) 26s ago 30m 16.5M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (3m) 80s ago 34m 45.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (3m) 80s ago 33m 53.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (2m) 80s ago 33m 42.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (85s) 80s ago 32m 14.3M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (50s) 26s ago 32m 46.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (31s) 26s ago 32m 14.8M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (31m) 26s ago 31m 57.3M 3248M 17.2.0 e1d6a67b021e 9780cee91e76 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (31m) 26s ago 31m 64.0M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (8m) 26s ago 30m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (29m) 80s ago 29m 80.3M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:50:20.571 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (29m) 26s ago 29m 81.0M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:50:20.804 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:20 smithi111 ceph-mon[68099]: failed to encode map e143 with expected crc 2024-02-14T17:50:20.805 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:20 smithi111 ceph-mon[68099]: osdmap e143: 8 total, 7 up, 8 in 2024-02-14T17:50:20.805 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:20 smithi111 ceph-mon[68099]: failed to encode map e143 with expected crc 2024-02-14T17:50:20.805 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:20 smithi111 ceph-mon[68099]: from='client.44541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:20.805 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:20 smithi111 podman[87856]: 2024-02-14 17:50:20.607765165 +0000 UTC m=+0.458865054 container attach d7374f15561bc4a975c322163fd6238249e81d14811d99b6012b348ec7a97a5c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-14T17:50:20.805 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:20 smithi111 podman[87856]: 2024-02-14 17:50:20.804280368 +0000 UTC m=+0.655380225 container died d7374f15561bc4a975c322163fd6238249e81d14811d99b6012b348ec7a97a5c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[88777]: failed to encode map e143 with expected crc 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[88777]: osdmap e143: 8 total, 7 up, 8 in 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[88777]: failed to encode map e143 with expected crc 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[88777]: from='client.44541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[83585]: failed to encode map e143 with expected crc 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[83585]: osdmap e143: 8 total, 7 up, 8 in 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[83585]: failed to encode map e143 with expected crc 2024-02-14T17:50:20.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:20 smithi017 ceph-mon[83585]: from='client.44541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 6 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:20.975 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:50:20.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:50:20.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:20.976 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:50:20.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-14T17:50:20.976 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 11 2024-02-14T17:50:20.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:50:20.976 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "6/8 daemons upgraded", 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:50:21.318 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:50:21.666 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:21 smithi111 podman[87883]: 2024-02-14 17:50:21.445492905 +0000 UTC m=+0.635971575 container remove d7374f15561bc4a975c322163fd6238249e81d14811d99b6012b348ec7a97a5c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-02-14T17:50:21.666 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:21 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.6.service: Deactivated successfully. 2024-02-14T17:50:21.666 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:21 smithi111 systemd[1]: Stopped Ceph osd.6 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:50:21.667 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:21 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.6.service: Consumed 14.347s CPU time. 2024-02-14T17:50:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: from='client.54484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: from='client.44553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: pgmap v235: 161 pgs: 4 active+undersized+degraded, 15 peering, 6 stale+active+clean, 7 active+undersized, 129 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 12/627 objects degraded (1.914%) 2024-02-14T17:50:21.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-14T17:50:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 12/627 objects degraded (1.914%), 4 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1233372815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:21.985 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:21.985 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:21 smithi111 systemd[1]: Starting Ceph osd.6 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:50:21.985 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:21 smithi111 podman[87988]: 2024-02-14 17:50:21.785775701 +0000 UTC m=+0.017799353 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:50:21.985 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:21 smithi111 podman[87988]: 2024-02-14 17:50:21.896437052 +0000 UTC m=+0.128460693 container create a01c89cc7d7b488803486ae5bbdf8722fac265d069d95ba0ad08b512a1d87254 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:50:21] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.43.0" 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: from='client.54484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: from='client.44553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: pgmap v235: 161 pgs: 4 active+undersized+degraded, 15 peering, 6 stale+active+clean, 7 active+undersized, 129 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 12/627 objects degraded (1.914%) 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 12/627 objects degraded (1.914%), 4 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1233372815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:22.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: from='client.54484 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: from='client.44553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: pgmap v235: 161 pgs: 4 active+undersized+degraded, 15 peering, 6 stale+active+clean, 7 active+undersized, 129 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 12/627 objects degraded (1.914%) 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 12/627 objects degraded (1.914%), 4 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1233372815' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:22.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:22.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 podman[87988]: 2024-02-14 17:50:22.1245926 +0000 UTC m=+0.356616253 container init a01c89cc7d7b488803486ae5bbdf8722fac265d069d95ba0ad08b512a1d87254 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-02-14T17:50:22.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 podman[87988]: 2024-02-14 17:50:22.127852176 +0000 UTC m=+0.359875823 container start a01c89cc7d7b488803486ae5bbdf8722fac265d069d95ba0ad08b512a1d87254 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214) 2024-02-14T17:50:22.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 podman[87988]: 2024-02-14 17:50:22.199338298 +0000 UTC m=+0.431361941 container attach a01c89cc7d7b488803486ae5bbdf8722fac265d069d95ba0ad08b512a1d87254 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-14T17:50:22.912 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-mon[68099]: from='client.44565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:22.912 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate[88002]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-14T17:50:22.912 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate[88002]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-14T17:50:22.912 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 bash[87988]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-14T17:50:22.912 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 bash[87988]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-02-14T17:50:22.912 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 bash[87988]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 bash[87988]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 bash[87988]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 bash[87988]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 bash[87988]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate[88002]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate[88002]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate[88002]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate[88002]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate[88002]: --> ceph-volume raw activate successful for osd ID: 6 2024-02-14T17:50:22.913 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:22 smithi111 podman[87988]: 2024-02-14 17:50:22.588603689 +0000 UTC m=+0.820627352 container died a01c89cc7d7b488803486ae5bbdf8722fac265d069d95ba0ad08b512a1d87254 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-14T17:50:23.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:22 smithi017 ceph-mon[83585]: from='client.44565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:23.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:22 smithi017 ceph-mon[88777]: from='client.44565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:23.594 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 podman[88125]: 2024-02-14 17:50:23.295264478 +0000 UTC m=+0.701450612 container remove a01c89cc7d7b488803486ae5bbdf8722fac265d069d95ba0ad08b512a1d87254 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240214) 2024-02-14T17:50:23.594 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 podman[88190]: 2024-02-14 17:50:23.496204105 +0000 UTC m=+0.104161008 container create f5aa07b95e17a9eaca0d17b0a1617b9f2c03fcd820c32a46646f7ad5c3fcd6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-02-14T17:50:23.594 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 podman[88190]: 2024-02-14 17:50:23.409301868 +0000 UTC m=+0.017258772 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:50:23.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:23 smithi111 ceph-mon[68099]: pgmap v236: 161 pgs: 8 active+undersized+degraded, 15 peering, 4 stale+active+clean, 18 active+undersized, 116 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-02-14T17:50:23.984 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 podman[88190]: 2024-02-14 17:50:23.675007008 +0000 UTC m=+0.282963926 container init f5aa07b95e17a9eaca0d17b0a1617b9f2c03fcd820c32a46646f7ad5c3fcd6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-14T17:50:23.984 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 podman[88190]: 2024-02-14 17:50:23.678391724 +0000 UTC m=+0.286348629 container start f5aa07b95e17a9eaca0d17b0a1617b9f2c03fcd820c32a46646f7ad5c3fcd6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-14T17:50:23.984 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 ceph-osd[88209]: -- 172.21.15.111:0/869294660 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5621dc4c51e0 con 0x5621dc496c00 2024-02-14T17:50:23.984 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 bash[88190]: f5aa07b95e17a9eaca0d17b0a1617b9f2c03fcd820c32a46646f7ad5c3fcd6d4 2024-02-14T17:50:23.984 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:23 smithi111 systemd[1]: Started Ceph osd.6 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:50:24.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:23 smithi017 ceph-mon[83585]: pgmap v236: 161 pgs: 8 active+undersized+degraded, 15 peering, 4 stale+active+clean, 18 active+undersized, 116 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-02-14T17:50:24.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:23 smithi017 ceph-mon[88777]: pgmap v236: 161 pgs: 8 active+undersized+degraded, 15 peering, 4 stale+active+clean, 18 active+undersized, 116 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 36/627 objects degraded (5.742%) 2024-02-14T17:50:24.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:50:24.223+0000 7f031732e7c0 -1 Falling back to public interface 2024-02-14T17:50:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:24 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:24 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:24.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:24 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:24 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:24 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:24 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:24 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:24 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:24 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:26.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:25 smithi017 ceph-mon[83585]: pgmap v237: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-02-14T17:50:26.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:25 smithi017 ceph-mon[83585]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-14T17:50:26.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:25 smithi017 ceph-mon[88777]: pgmap v237: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-02-14T17:50:26.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:25 smithi017 ceph-mon[88777]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-14T17:50:26.208 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:25 smithi111 ceph-mon[68099]: pgmap v237: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-02-14T17:50:26.208 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:25 smithi111 ceph-mon[68099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-14T17:50:27.234 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:50:26.958+0000 7f031732e7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-02-14T17:50:27.234 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:50:26.983+0000 7f031732e7c0 -1 osd.6 141 log_to_monitors true 2024-02-14T17:50:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[83585]: pgmap v238: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-02-14T17:50:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[83585]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:50:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[83585]: from='osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:50:28.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[88777]: pgmap v238: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-02-14T17:50:28.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[88777]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:50:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[88777]: from='osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:50:28.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:27 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:28.233 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:50:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:50:27.849+0000 7f0308786700 -1 osd.6 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:50:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:27 smithi111 ceph-mon[68099]: pgmap v238: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-02-14T17:50:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:27 smithi111 ceph-mon[68099]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:50:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:27 smithi111 ceph-mon[68099]: from='osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-14T17:50:28.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:27 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 18 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: osdmap e144: 8 total, 7 up, 8 in 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: from='osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: failed to encode map e144 with expected crc 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:29.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: osdmap e144: 8 total, 7 up, 8 in 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: from='osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: failed to encode map e144 with expected crc 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:29.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:28 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:29.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-14T17:50:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: osdmap e144: 8 total, 7 up, 8 in 2024-02-14T17:50:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: from='osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:50:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:50:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: failed to encode map e144 with expected crc 2024-02-14T17:50:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:29.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:28 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:30.144 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: pgmap v240: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s; 74/627 objects degraded (11.802%) 2024-02-14T17:50:30.144 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:50:30.144 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.144 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145] boot 2024-02-14T17:50:30.144 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: osdmap e145: 8 total, 8 up, 8 in 2024-02-14T17:50:30.144 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:50:30.144 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:30.145 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:29 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: pgmap v240: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s; 74/627 objects degraded (11.802%) 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145] boot 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: osdmap e145: 8 total, 8 up, 8 in 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: pgmap v240: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 114 B/s rd, 0 op/s; 74/627 objects degraded (11.802%) 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: osd.6 [v2:172.21.15.111:6816/2304781145,v1:172.21.15.111:6817/2304781145] boot 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: osdmap e145: 8 total, 8 up, 8 in 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: failed to encode map e145 with expected crc 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:30.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:29 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:31.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:30 smithi111 ceph-mon[68099]: failed to encode map e146 with expected crc 2024-02-14T17:50:31.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:30 smithi111 ceph-mon[68099]: osdmap e146: 8 total, 8 up, 8 in 2024-02-14T17:50:31.116 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:30 smithi111 ceph-mon[68099]: failed to encode map e146 with expected crc 2024-02-14T17:50:31.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:30 smithi017 ceph-mon[83585]: failed to encode map e146 with expected crc 2024-02-14T17:50:31.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:30 smithi017 ceph-mon[83585]: osdmap e146: 8 total, 8 up, 8 in 2024-02-14T17:50:31.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:30 smithi017 ceph-mon[83585]: failed to encode map e146 with expected crc 2024-02-14T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:30 smithi017 ceph-mon[88777]: failed to encode map e146 with expected crc 2024-02-14T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:30 smithi017 ceph-mon[88777]: osdmap e146: 8 total, 8 up, 8 in 2024-02-14T17:50:31.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:30 smithi017 ceph-mon[88777]: failed to encode map e146 with expected crc 2024-02-14T17:50:32.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:31 smithi017 ceph-mon[83585]: pgmap v243: 161 pgs: 19 peering, 11 active+undersized+degraded, 23 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/627 objects degraded (7.656%) 2024-02-14T17:50:32.091 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:31 smithi017 ceph-mon[88777]: pgmap v243: 161 pgs: 19 peering, 11 active+undersized+degraded, 23 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/627 objects degraded (7.656%) 2024-02-14T17:50:32.091 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:50:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:50:31] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.43.0" 2024-02-14T17:50:32.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:31 smithi111 ceph-mon[68099]: pgmap v243: 161 pgs: 19 peering, 11 active+undersized+degraded, 23 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/627 objects degraded (7.656%) 2024-02-14T17:50:33.172 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 48/627 objects degraded (7.656%), 11 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.173 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:32 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:33.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:50:32.812+0000 7f1ca1663700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-14T17:50:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 48/627 objects degraded (7.656%), 11 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:33.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:50:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:33.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 48/627 objects degraded (7.656%), 11 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:33.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:33.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:50:33.192 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:33.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:33.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:33.193 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:32 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:34.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[88777]: pgmap v244: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[88777]: pgmap v245: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[88777]: pgmap v246: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 761 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:34.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[88777]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-14T17:50:34.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[83585]: pgmap v244: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[83585]: pgmap v245: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[83585]: pgmap v246: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 761 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:34.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:33 smithi017 ceph-mon[83585]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-14T17:50:34.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:33 smithi111 ceph-mon[68099]: pgmap v244: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:33 smithi111 ceph-mon[68099]: pgmap v245: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 633 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:33 smithi111 ceph-mon[68099]: pgmap v246: 161 pgs: 19 peering, 7 active+undersized+degraded, 14 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 761 B/s rd, 0 op/s; 24/627 objects degraded (3.828%) 2024-02-14T17:50:34.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:33 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:34.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:33 smithi111 ceph-mon[68099]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-02-14T17:50:35.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:34 smithi017 ceph-mon[88777]: pgmap v247: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 525 B/s rd, 0 op/s 2024-02-14T17:50:35.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:34 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 7 pgs degraded) 2024-02-14T17:50:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:34 smithi017 ceph-mon[83585]: pgmap v247: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 525 B/s rd, 0 op/s 2024-02-14T17:50:35.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:34 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 7 pgs degraded) 2024-02-14T17:50:35.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:34 smithi111 ceph-mon[68099]: pgmap v247: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 525 B/s rd, 0 op/s 2024-02-14T17:50:35.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:34 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/627 objects degraded (3.828%), 7 pgs degraded) 2024-02-14T17:50:37.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:37 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:37.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:37 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:37.443 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:37 smithi017 ceph-mon[83585]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:37.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:37 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:37.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:37 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:37.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:37 smithi017 ceph-mon[88777]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:37.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:37 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:37.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:37 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:37.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:37 smithi111 ceph-mon[68099]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:38.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:38.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:38.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:39.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:39 smithi017 ceph-mon[83585]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-02-14T17:50:39.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:39 smithi017 ceph-mon[88777]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-02-14T17:50:39.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:39 smithi111 ceph-mon[68099]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-02-14T17:50:41.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:41 smithi111 ceph-mon[68099]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:42.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:41 smithi017 ceph-mon[83585]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:42.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:41 smithi017 ceph-mon[88777]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:50:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:50:41] "GET /metrics HTTP/1.1" 200 38287 "" "Prometheus/2.43.0" 2024-02-14T17:50:43.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:43 smithi111 ceph-mon[68099]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:50:44.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:43 smithi017 ceph-mon[83585]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:50:44.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:43 smithi017 ceph-mon[88777]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:50:45.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:45 smithi111 ceph-mon[68099]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:46.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:45 smithi017 ceph-mon[83585]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:46.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:45 smithi017 ceph-mon[88777]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:47.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:47 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:47.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:47 smithi017 ceph-mon[83585]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:47.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:47 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:47.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:47 smithi017 ceph-mon[88777]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:47.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:47 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:47.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:47 smithi111 ceph-mon[68099]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:48.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:48.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:48 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:48.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:48 smithi111 ceph-mon[68099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:48.395 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:48 smithi111 ceph-mon[68099]: Upgrade: osd.7 is safe to restart 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[83585]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[83585]: Upgrade: osd.7 is safe to restart 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[88777]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-02-14T17:50:48.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:48 smithi017 ceph-mon[88777]: Upgrade: osd.7 is safe to restart 2024-02-14T17:50:49.428 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:49 smithi111 ceph-mon[68099]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:49.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:49 smithi017 ceph-mon[83585]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:49.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:49 smithi017 ceph-mon[88777]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:50.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:50 smithi111 ceph-mon[68099]: Upgrade: Updating osd.7 2024-02-14T17:50:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:50 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:50 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:50:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:50 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:50.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:50 smithi111 ceph-mon[68099]: Deploying daemon osd.7 on smithi111 2024-02-14T17:50:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[83585]: Upgrade: Updating osd.7 2024-02-14T17:50:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:50:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[83585]: Deploying daemon osd.7 on smithi111 2024-02-14T17:50:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[88777]: Upgrade: Updating osd.7 2024-02-14T17:50:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-14T17:50:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:50:50.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:50 smithi017 ceph-mon[88777]: Deploying daemon osd.7 on smithi111 2024-02-14T17:50:51.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:51 smithi111 ceph-mon[68099]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:51.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:51.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:51 smithi017 ceph-mon[83585]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:51.683 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:51.684 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:51 smithi017 ceph-mon[88777]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:50:51.684 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:50:51.684 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:50:51.939 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:50:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:50:51] "GET /metrics HTTP/1.1" 200 38288 "" "Prometheus/2.43.0" 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (8m) 112s ago 30m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (8m) 24s ago 30m 89.3M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 112s ago 29m 61.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (9m) 24s ago 34m 452M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (19m) 112s ago 36m 507M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (6m) 112s ago 36m 53.6M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (7m) 24s ago 35m 47.2M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:50:52.346 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (6m) 112s ago 35m 39.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (9m) 112s ago 30m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (9m) 24s ago 30m 16.7M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (4m) 112s ago 34m 45.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (3m) 112s ago 34m 53.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (2m) 112s ago 33m 42.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (117s) 112s ago 33m 14.3M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (82s) 24s ago 33m 47.4M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (62s) 24s ago 32m 53.0M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (28s) 24s ago 32m 12.0M 3248M 19.0.0-1430-g8727ce93 170846cf6432 f5aa07b95e17 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (31m) 24s ago 31m 65.7M 3248M 17.2.0 e1d6a67b021e 248e75a794bc 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (8m) 24s ago 31m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (29m) 112s ago 29m 80.3M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:50:52.347 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (29m) 24s ago 29m 81.2M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:50:52.435 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:52 smithi111 ceph-mon[68099]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:52.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:52 smithi017 ceph-mon[83585]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:52 smithi017 ceph-mon[88777]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:52.733 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:52 smithi111 systemd[1]: Stopping Ceph osd.7 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:50:52.734 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[37812]: 2024-02-14T17:50:52.672+0000 7ff37476b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:50:52.734 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[37812]: 2024-02-14T17:50:52.672+0000 7ff37476b700 -1 osd.7 146 *** Got signal Terminated *** 2024-02-14T17:50:52.734 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[37812]: 2024-02-14T17:50:52.672+0000 7ff37476b700 -1 osd.7 146 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:50:52.750 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 7 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:50:52.751 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:50:52.752 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-02-14T17:50:52.752 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 12 2024-02-14T17:50:52.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:50:52.752 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:50:53.096 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:50:53.096 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:50:53.096 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:50:53.097 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-02-14T17:50:53.097 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:50:53.097 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "7/8 daemons upgraded", 2024-02-14T17:50:53.097 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading osd daemons", 2024-02-14T17:50:53.097 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:50:53.097 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:50:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[83585]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[83585]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[83585]: osd.7 marked itself down and dead 2024-02-14T17:50:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[83585]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:53.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2680718230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[88777]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:53.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[88777]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:53.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[88777]: osd.7 marked itself down and dead 2024-02-14T17:50:53.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[88777]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:53.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:53 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2680718230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:53.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:53 smithi111 ceph-mon[68099]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:53 smithi111 ceph-mon[68099]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:53 smithi111 ceph-mon[68099]: osd.7 marked itself down and dead 2024-02-14T17:50:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:53 smithi111 ceph-mon[68099]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:50:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:53 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2680718230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:50:54.083 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:53 smithi111 podman[92434]: 2024-02-14 17:50:53.797223596 +0000 UTC m=+1.222327101 container died 248e75a794bcfaf2d27921959cea106860966ad13b192d710e739b68247d2b0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, architecture=x86_64, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-02-14T17:50:54.424 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:54 smithi111 ceph-mon[68099]: from='client.44601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:54.425 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:54 smithi111 ceph-mon[68099]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:50:54.425 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:54 smithi111 ceph-mon[68099]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-14T17:50:54.425 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:54 smithi111 ceph-mon[68099]: osdmap e147: 8 total, 7 up, 8 in 2024-02-14T17:50:54.425 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:54 smithi111 podman[92434]: 2024-02-14 17:50:54.181343028 +0000 UTC m=+1.606446544 container cleanup 248e75a794bcfaf2d27921959cea106860966ad13b192d710e739b68247d2b0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, version=8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, architecture=x86_64, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, name=centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc.) 2024-02-14T17:50:54.425 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:54 smithi111 bash[92434]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7 2024-02-14T17:50:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[83585]: from='client.44601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[83585]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:50:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[83585]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-14T17:50:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[83585]: osdmap e147: 8 total, 7 up, 8 in 2024-02-14T17:50:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[88777]: from='client.44601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:50:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[88777]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-14T17:50:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[88777]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-02-14T17:50:54.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:54 smithi017 ceph-mon[88777]: osdmap e147: 8 total, 7 up, 8 in 2024-02-14T17:50:54.733 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:54 smithi111 podman[92451]: 2024-02-14 17:50:54.424201377 +0000 UTC m=+0.621224769 container remove 248e75a794bcfaf2d27921959cea106860966ad13b192d710e739b68247d2b0a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, release=754, io.openshift.expose-services=, version=8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc.) 2024-02-14T17:50:55.044 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:54 smithi111 podman[92542]: 2024-02-14 17:50:54.651898036 +0000 UTC m=+0.017896912 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:50:55.044 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:54 smithi111 podman[92542]: 2024-02-14 17:50:54.758528893 +0000 UTC m=+0.124527768 container create fde3deaef50d587409f8365861cc0cf7042808b2b10be97038f27d9ff462926a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240214, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-14T17:50:55.045 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:54 smithi111 podman[92542]: 2024-02-14 17:50:54.969518634 +0000 UTC m=+0.335517515 container init fde3deaef50d587409f8365861cc0cf7042808b2b10be97038f27d9ff462926a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True) 2024-02-14T17:50:55.045 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:54 smithi111 podman[92542]: 2024-02-14 17:50:54.972900535 +0000 UTC m=+0.338899413 container start fde3deaef50d587409f8365861cc0cf7042808b2b10be97038f27d9ff462926a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-02-14T17:50:55.045 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:55 smithi111 podman[92542]: 2024-02-14 17:50:55.044240644 +0000 UTC m=+0.410239537 container attach fde3deaef50d587409f8365861cc0cf7042808b2b10be97038f27d9ff462926a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:50:55.325 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:55 smithi111 podman[92542]: 2024-02-14 17:50:55.264599662 +0000 UTC m=+0.630598528 container died fde3deaef50d587409f8365861cc0cf7042808b2b10be97038f27d9ff462926a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-02-14T17:50:55.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:55 smithi017 ceph-mon[83585]: osdmap e148: 8 total, 7 up, 8 in 2024-02-14T17:50:55.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:55 smithi017 ceph-mon[83585]: pgmap v259: 161 pgs: 24 peering, 13 stale+active+clean, 124 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:50:55.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:55 smithi017 ceph-mon[88777]: osdmap e148: 8 total, 7 up, 8 in 2024-02-14T17:50:55.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:55 smithi017 ceph-mon[88777]: pgmap v259: 161 pgs: 24 peering, 13 stale+active+clean, 124 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:50:55.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:55 smithi111 ceph-mon[68099]: osdmap e148: 8 total, 7 up, 8 in 2024-02-14T17:50:55.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:55 smithi111 ceph-mon[68099]: pgmap v259: 161 pgs: 24 peering, 13 stale+active+clean, 124 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:50:56.208 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:55 smithi111 podman[92568]: 2024-02-14 17:50:55.798345205 +0000 UTC m=+0.524078651 container remove fde3deaef50d587409f8365861cc0cf7042808b2b10be97038f27d9ff462926a (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:50:56.208 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:55 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-14T17:50:56.208 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:55 smithi111 systemd[1]: Stopped Ceph osd.7 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:50:56.208 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:55 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.7.service: Consumed 16.697s CPU time. 2024-02-14T17:50:56.208 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 systemd[1]: Starting Ceph osd.7 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:50:56.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-mon[68099]: Health check failed: Reduced data availability: 2 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-02-14T17:50:56.484 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 podman[92669]: 2024-02-14 17:50:56.207542977 +0000 UTC m=+0.095129900 container create 4e86589c099f523b548ca95f834e7acab153ed603dfe35d48b80f4fde10495b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:50:56.484 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 podman[92669]: 2024-02-14 17:50:56.129789072 +0000 UTC m=+0.017376003 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:50:56.484 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 podman[92669]: 2024-02-14 17:50:56.443438998 +0000 UTC m=+0.331025924 container init 4e86589c099f523b548ca95f834e7acab153ed603dfe35d48b80f4fde10495b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-02-14T17:50:56.484 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 podman[92669]: 2024-02-14 17:50:56.446964529 +0000 UTC m=+0.334551455 container start 4e86589c099f523b548ca95f834e7acab153ed603dfe35d48b80f4fde10495b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-14T17:50:56.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:56 smithi017 ceph-mon[83585]: Health check failed: Reduced data availability: 2 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-02-14T17:50:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:56 smithi017 ceph-mon[88777]: Health check failed: Reduced data availability: 2 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-02-14T17:50:56.890 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 podman[92669]: 2024-02-14 17:50:56.518367891 +0000 UTC m=+0.405954833 container attach 4e86589c099f523b548ca95f834e7acab153ed603dfe35d48b80f4fde10495b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5) 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate[92684]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate[92684]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 bash[92669]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 bash[92669]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 bash[92669]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 bash[92669]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 bash[92669]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 bash[92669]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 bash[92669]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate[92684]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-02-14T17:50:57.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate[92684]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-02-14T17:50:57.235 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate[92684]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-02-14T17:50:57.235 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate[92684]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-02-14T17:50:57.235 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate[92684]: --> ceph-volume raw activate successful for osd ID: 7 2024-02-14T17:50:57.235 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:56 smithi111 podman[92669]: 2024-02-14 17:50:56.914854927 +0000 UTC m=+0.802441842 container died 4e86589c099f523b548ca95f834e7acab153ed603dfe35d48b80f4fde10495b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-14T17:50:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:57 smithi017 ceph-mon[88777]: pgmap v260: 161 pgs: 8 active+undersized+degraded, 24 peering, 11 stale+active+clean, 9 active+undersized, 109 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 21/627 objects degraded (3.349%) 2024-02-14T17:50:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:57 smithi017 ceph-mon[83585]: pgmap v260: 161 pgs: 8 active+undersized+degraded, 24 peering, 11 stale+active+clean, 9 active+undersized, 109 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 21/627 objects degraded (3.349%) 2024-02-14T17:50:57.692 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:57 smithi111 ceph-mon[68099]: pgmap v260: 161 pgs: 8 active+undersized+degraded, 24 peering, 11 stale+active+clean, 9 active+undersized, 109 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 21/627 objects degraded (3.349%) 2024-02-14T17:50:57.692 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 podman[92816]: 2024-02-14 17:50:57.480716059 +0000 UTC m=+0.560250298 container remove 4e86589c099f523b548ca95f834e7acab153ed603dfe35d48b80f4fde10495b5 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:50:57.967 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 podman[92879]: 2024-02-14 17:50:57.595368781 +0000 UTC m=+0.017853473 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:50:57.968 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 podman[92879]: 2024-02-14 17:50:57.698359551 +0000 UTC m=+0.120844238 container create 3685781bd0e6b63465428be9b01c8abb4805f51772f8d2579de83c4ed7ac79d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, CEPH_POINT_RELEASE=) 2024-02-14T17:50:57.968 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 podman[92879]: 2024-02-14 17:50:57.909012146 +0000 UTC m=+0.331496833 container init 3685781bd0e6b63465428be9b01c8abb4805f51772f8d2579de83c4ed7ac79d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-14T17:50:57.968 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 podman[92879]: 2024-02-14 17:50:57.912195847 +0000 UTC m=+0.334680535 container start 3685781bd0e6b63465428be9b01c8abb4805f51772f8d2579de83c4ed7ac79d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD) 2024-02-14T17:50:57.968 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 ceph-osd[92898]: -- 172.21.15.111:0/1074947101 <== mon.2 v2:172.21.15.111:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556060d431e0 con 0x556060d15000 2024-02-14T17:50:57.968 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 bash[92879]: 3685781bd0e6b63465428be9b01c8abb4805f51772f8d2579de83c4ed7ac79d6 2024-02-14T17:50:58.233 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:57 smithi111 systemd[1]: Started Ceph osd.7 for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[83585]: Health check failed: Degraded data redundancy: 21/627 objects degraded (3.349%), 8 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[88777]: Health check failed: Degraded data redundancy: 21/627 objects degraded (3.349%), 8 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:58.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:50:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:58 smithi111 ceph-mon[68099]: Health check failed: Degraded data redundancy: 21/627 objects degraded (3.349%), 8 pgs degraded (PG_DEGRADED) 2024-02-14T17:50:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:50:58.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:50:58.734 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:50:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:50:58.457+0000 7fed60bb97c0 -1 Falling back to public interface 2024-02-14T17:50:59.631 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:50:59 smithi111 ceph-mon[68099]: pgmap v261: 161 pgs: 20 active+undersized+degraded, 11 peering, 38 active+undersized, 92 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 49/627 objects degraded (7.815%) 2024-02-14T17:50:59.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:50:59 smithi017 ceph-mon[83585]: pgmap v261: 161 pgs: 20 active+undersized+degraded, 11 peering, 38 active+undersized, 92 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 49/627 objects degraded (7.815%) 2024-02-14T17:50:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:50:59 smithi017 ceph-mon[88777]: pgmap v261: 161 pgs: 20 active+undersized+degraded, 11 peering, 38 active+undersized, 92 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 49/627 objects degraded (7.815%) 2024-02-14T17:51:01.155 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:00 smithi111 ceph-mon[68099]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 6 pgs peering) 2024-02-14T17:51:01.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:00 smithi017 ceph-mon[83585]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 6 pgs peering) 2024-02-14T17:51:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:00 smithi017 ceph-mon[88777]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 6 pgs peering) 2024-02-14T17:51:01.483 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:51:01 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:51:01.203+0000 7fed60bb97c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-02-14T17:51:01.484 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:51:01 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:51:01.232+0000 7fed60bb97c0 -1 osd.7 146 log_to_monitors true 2024-02-14T17:51:02.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:01 smithi017 ceph-mon[83585]: pgmap v262: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-02-14T17:51:02.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:01 smithi017 ceph-mon[83585]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:51:02.091 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:01 smithi017 ceph-mon[83585]: from='osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:51:02.092 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:01 smithi017 ceph-mon[88777]: pgmap v262: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-02-14T17:51:02.092 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:01 smithi017 ceph-mon[88777]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:51:02.092 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:01 smithi017 ceph-mon[88777]: from='osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:51:02.092 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:51:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:51:01] "GET /metrics HTTP/1.1" 200 38288 "" "Prometheus/2.43.0" 2024-02-14T17:51:02.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:01 smithi111 ceph-mon[68099]: pgmap v262: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-02-14T17:51:02.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:01 smithi111 ceph-mon[68099]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:51:02.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:01 smithi111 ceph-mon[68099]: from='osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-14T17:51:02.733 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:51:02.256+0000 7fed4e009700 -1 osd.7 146 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: osdmap e149: 8 total, 7 up, 8 in 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: from='osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: from='osd.7 ' entity='osd.7' 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 28 pgs degraded (PG_DEGRADED) 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: osdmap e149: 8 total, 7 up, 8 in 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: from='osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: from='osd.7 ' entity='osd.7' 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:02 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 28 pgs degraded (PG_DEGRADED) 2024-02-14T17:51:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-14T17:51:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: osdmap e149: 8 total, 7 up, 8 in 2024-02-14T17:51:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: from='osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:51:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-02-14T17:51:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: from='osd.7 ' entity='osd.7' 2024-02-14T17:51:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:02 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 28 pgs degraded (PG_DEGRADED) 2024-02-14T17:51:04.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:03 smithi111 ceph-mon[68099]: pgmap v264: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 367 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-02-14T17:51:04.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:03 smithi111 ceph-mon[68099]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:51:04.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:03 smithi111 ceph-mon[68099]: osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935] boot 2024-02-14T17:51:04.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:03 smithi111 ceph-mon[68099]: osdmap e150: 8 total, 8 up, 8 in 2024-02-14T17:51:04.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:51:04.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:04.063 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:03 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[83585]: pgmap v264: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 367 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[83585]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[83585]: osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935] boot 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[83585]: osdmap e150: 8 total, 8 up, 8 in 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[88777]: pgmap v264: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 367 B/s rd, 0 op/s; 83/627 objects degraded (13.238%) 2024-02-14T17:51:04.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[88777]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-14T17:51:04.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[88777]: osd.7 [v2:172.21.15.111:6824/80892935,v1:172.21.15.111:6825/80892935] boot 2024-02-14T17:51:04.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[88777]: osdmap e150: 8 total, 8 up, 8 in 2024-02-14T17:51:04.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-14T17:51:04.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:04.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:03 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:05.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:05 smithi111 ceph-mon[68099]: osdmap e151: 8 total, 8 up, 8 in 2024-02-14T17:51:05.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:05 smithi111 ceph-mon[68099]: pgmap v267: 161 pgs: 14 peering, 21 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-02-14T17:51:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:05 smithi017 ceph-mon[83585]: osdmap e151: 8 total, 8 up, 8 in 2024-02-14T17:51:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:05 smithi017 ceph-mon[83585]: pgmap v267: 161 pgs: 14 peering, 21 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-02-14T17:51:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:05 smithi017 ceph-mon[88777]: osdmap e151: 8 total, 8 up, 8 in 2024-02-14T17:51:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:05 smithi017 ceph-mon[88777]: pgmap v267: 161 pgs: 14 peering, 21 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/627 objects degraded (10.207%) 2024-02-14T17:51:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: pgmap v268: 161 pgs: 14 peering, 16 active+undersized+degraded, 27 active+undersized, 104 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s; 49/627 objects degraded (7.815%) 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all osd 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-14T17:51:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.442 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: pgmap v268: 161 pgs: 14 peering, 16 active+undersized+degraded, 27 active+undersized, 104 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s; 49/627 objects degraded (7.815%) 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all osd 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-14T17:51:07.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-14T17:51:07.444 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: Upgrade: Setting require_osd_release to 19 squid 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: Upgrade: Setting require_osd_release to 19 squid 2024-02-14T17:51:07.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: pgmap v268: 161 pgs: 14 peering, 16 active+undersized+degraded, 27 active+undersized, 104 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s; 49/627 objects degraded (7.815%) 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all osd 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-02-14T17:51:07.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-02-14T17:51:07.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-02-14T17:51:07.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-02-14T17:51:07.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-02-14T17:51:07.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-02-14T17:51:07.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: Upgrade: Setting require_osd_release to 19 squid 2024-02-14T17:51:07.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-02-14T17:51:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Health check update: Degraded data redundancy: 64/627 objects degraded (10.207%), 21 pgs degraded (PG_DEGRADED) 2024-02-14T17:51:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-14T17:51:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: osdmap e152: 8 total, 8 up, 8 in 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mds 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nfs 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Upgrade: Finalizing container_image settings 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:51:08.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Health check update: Degraded data redundancy: 64/627 objects degraded (10.207%), 21 pgs degraded (PG_DEGRADED) 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: osdmap e152: 8 total, 8 up, 8 in 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mds 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:51:08.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nfs 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Upgrade: Finalizing container_image settings 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-14T17:51:08.445 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.446 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-14T17:51:08.447 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.448 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: Upgrade: Complete! 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: pgmap v270: 161 pgs: 14 peering, 4 active+undersized+degraded, 7 active+undersized, 136 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 55 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: pgmap v271: 161 pgs: 14 peering, 4 active+undersized+degraded, 7 active+undersized, 136 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:08.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: Upgrade: Complete! 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: pgmap v270: 161 pgs: 14 peering, 4 active+undersized+degraded, 7 active+undersized, 136 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 55 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: pgmap v271: 161 pgs: 14 peering, 4 active+undersized+degraded, 7 active+undersized, 136 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:08.450 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:08.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:08.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:08.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:08 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Health check update: Degraded data redundancy: 64/627 objects degraded (10.207%), 21 pgs degraded (PG_DEGRADED) 2024-02-14T17:51:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-02-14T17:51:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-02-14T17:51:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: osdmap e152: 8 total, 8 up, 8 in 2024-02-14T17:51:08.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mds 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nfs 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:51:08.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Upgrade: Finalizing container_image settings 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-14T17:51:08.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: Upgrade: Complete! 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:08.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:08.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: pgmap v270: 161 pgs: 14 peering, 4 active+undersized+degraded, 7 active+undersized, 136 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 55 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:08.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: pgmap v271: 161 pgs: 14 peering, 4 active+undersized+degraded, 7 active+undersized, 136 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 62 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:08.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:08.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:08.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:08.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:08.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:08 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:10.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:09 smithi017 ceph-mon[83585]: pgmap v272: 161 pgs: 14 peering, 4 active+undersized+degraded, 3 active+undersized, 140 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 76 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:10.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:09 smithi017 ceph-mon[88777]: pgmap v272: 161 pgs: 14 peering, 4 active+undersized+degraded, 3 active+undersized, 140 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 76 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:10.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:09 smithi111 ceph-mon[68099]: pgmap v272: 161 pgs: 14 peering, 4 active+undersized+degraded, 3 active+undersized, 140 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 76 op/s; 7/627 objects degraded (1.116%) 2024-02-14T17:51:11.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:10 smithi017 ceph-mon[83585]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 4 pgs degraded) 2024-02-14T17:51:11.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:10 smithi017 ceph-mon[88777]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 4 pgs degraded) 2024-02-14T17:51:11.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:10 smithi111 ceph-mon[68099]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 4 pgs degraded) 2024-02-14T17:51:12.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:11 smithi017 ceph-mon[88777]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-02-14T17:51:12.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:11 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:12.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:11 smithi017 ceph-mon[83585]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-02-14T17:51:12.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:11 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:12.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:51:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:51:11] "GET /metrics HTTP/1.1" 200 38234 "" "Prometheus/2.43.0" 2024-02-14T17:51:12.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:11 smithi111 ceph-mon[68099]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 0 B/s wr, 137 op/s 2024-02-14T17:51:12.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:11 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:14.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:13 smithi017 ceph-mon[83585]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-02-14T17:51:14.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:13 smithi017 ceph-mon[88777]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-02-14T17:51:14.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:13 smithi111 ceph-mon[68099]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 139 op/s 2024-02-14T17:51:15.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:14 smithi111 ceph-mon[68099]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-02-14T17:51:15.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:14 smithi017 ceph-mon[83585]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-02-14T17:51:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:14 smithi017 ceph-mon[88777]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 143 op/s 2024-02-14T17:51:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:17 smithi017 ceph-mon[83585]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-02-14T17:51:17.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:17 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:17.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:17 smithi017 ceph-mon[88777]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-02-14T17:51:17.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:17 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:17.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:17 smithi111 ceph-mon[68099]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 118 op/s 2024-02-14T17:51:17.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:17 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:18.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:18.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:18.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:19.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:19 smithi017 ceph-mon[83585]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-02-14T17:51:19.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:19 smithi017 ceph-mon[88777]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-02-14T17:51:19.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:19 smithi111 ceph-mon[68099]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 114 op/s 2024-02-14T17:51:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:21 smithi017 ceph-mon[83585]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-02-14T17:51:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:21 smithi017 ceph-mon[88777]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-02-14T17:51:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:21.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:21 smithi111 ceph-mon[68099]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 97 op/s 2024-02-14T17:51:21.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:21.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:51:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:51:21] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-02-14T17:51:23.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:23 smithi017 ceph-mon[83585]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:51:23.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:23 smithi017 ceph-mon[88777]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:51:23.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:23 smithi111 ceph-mon[68099]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-02-14T17:51:24.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:24 smithi017 ceph-mon[83585]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:24.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:24 smithi017 ceph-mon[88777]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:24.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:51:24.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:24 smithi111 ceph-mon[68099]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:25.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:25 smithi017 ceph-mon[83585]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-02-14T17:51:25.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:25 smithi017 ceph-mon[88777]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-02-14T17:51:25.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:25 smithi111 ceph-mon[68099]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 57 op/s 2024-02-14T17:51:26.436 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (9m) 2m ago 31m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (8m) 24s ago 30m 87.3M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 2m ago 30m 61.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (10m) 24s ago 35m 452M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (20m) 2m ago 36m 507M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (6m) 2m ago 37m 53.6M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (7m) 24s ago 35m 48.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (6m) 2m ago 35m 39.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (9m) 2m ago 31m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:51:26.437 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (9m) 24s ago 31m 17.0M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (4m) 2m ago 35m 45.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (4m) 2m ago 34m 53.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (3m) 2m ago 34m 42.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (2m) 2m ago 33m 14.3M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (116s) 24s ago 33m 49.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (96s) 24s ago 33m 53.9M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (63s) 24s ago 32m 41.9M 3248M 19.0.0-1430-g8727ce93 170846cf6432 f5aa07b95e17 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (28s) 24s ago 32m 12.6M 3248M 19.0.0-1430-g8727ce93 170846cf6432 3685781bd0e6 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (9m) 24s ago 31m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (30m) 2m ago 30m 80.3M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:51:26.438 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (30m) 24s ago 30m 81.4M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:51:27.132 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-02-14T17:51:27.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:27 smithi017 ceph-mon[83585]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-02-14T17:51:27.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:27 smithi017 ceph-mon[83585]: from='client.44616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:27 smithi017 ceph-mon[88777]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-02-14T17:51:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:27 smithi017 ceph-mon[88777]: from='client.44616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:27.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:27 smithi111 ceph-mon[68099]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 27 op/s 2024-02-14T17:51:27.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:27 smithi111 ceph-mon[68099]: from='client.44616 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:28.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:28.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:28.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:51:29.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:29 smithi017 ceph-mon[83585]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:51:29.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:29 smithi017 ceph-mon[88777]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:51:29.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:29 smithi111 ceph-mon[68099]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-14T17:51:30.000 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-02-14T17:51:30.309 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:30 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2466234828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:30.310 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:30 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2466234828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:30.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:30 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2466234828' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:31.307 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:31 smithi017 ceph-mon[83585]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:31.307 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:31 smithi017 ceph-mon[88777]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:31.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:31 smithi111 ceph-mon[68099]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:31.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:51:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:51:31] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-02-14T17:51:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)" 2024-02-14T17:51:32.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:32 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2405771654' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:32.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:32 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2405771654' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:32.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:32 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2405771654' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:32.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:51:33.392 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:33 smithi017 ceph-mon[83585]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:33.393 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:33 smithi017 ceph-mon[88777]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:33.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:33 smithi111 ceph-mon[68099]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": null, 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": false, 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "", 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": null, 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:51:34.991 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:51:35.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:35 smithi017 ceph-mon[83585]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:35.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:35 smithi017 ceph-mon[88777]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:35.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:35 smithi111 ceph-mon[68099]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:35.739 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:51:36.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:36 smithi017 ceph-mon[83585]: from='client.54574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:36.304 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:36.305 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:36 smithi017 ceph-mon[88777]: from='client.54574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:36.305 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:36.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:36 smithi111 ceph-mon[68099]: from='client.54574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:36.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:37.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:37 smithi017 ceph-mon[83585]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:37.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:37 smithi017 ceph-mon[88777]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:37.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:37 smithi111 ceph-mon[68099]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:37.688 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:51:37.688 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:51:37.688 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:51:38.328 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:38.329 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:38 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/718904683' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:51:38.329 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:38.329 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:38 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/718904683' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:51:38.353 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-02-14T17:51:38.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:38.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:38 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/718904683' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:51:39.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:39 smithi111 ceph-mon[68099]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:39.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:39 smithi017 ceph-mon[83585]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:39.545 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:39 smithi017 ceph-mon[88777]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:41 smithi017 ceph-mon[83585]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:41.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:41 smithi017 ceph-mon[83585]: from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:41.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:41 smithi017 ceph-mon[88777]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:41.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:41 smithi017 ceph-mon[88777]: from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:41.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:41 smithi111 ceph-mon[68099]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:51:41.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:41 smithi111 ceph-mon[68099]: from='client.44646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:51:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:51:41] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-02-14T17:51:42.362 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:51:43.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-14T17:51:43.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[83585]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:43.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[83585]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:51:43.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[88777]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[88777]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:43.603 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:43 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:43.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:43 smithi111 ceph-mon[68099]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:51:43.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:43 smithi111 ceph-mon[68099]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:51:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:43 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:43 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:43 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:43 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:51:43.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:43 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:44.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:44 smithi017 ceph-mon[83585]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-02-14T17:51:44.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:44 smithi017 ceph-mon[83585]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:44.623 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:44 smithi017 ceph-mon[83585]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:51:44.623 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:44 smithi017 ceph-mon[88777]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-02-14T17:51:44.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:44 smithi017 ceph-mon[88777]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:44.624 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:44 smithi017 ceph-mon[88777]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:51:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:44 smithi111 ceph-mon[68099]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-02-14T17:51:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:44 smithi111 ceph-mon[68099]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:44.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:44 smithi111 ceph-mon[68099]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:51:45.211 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:51:45.888 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (9m) 2m ago 31m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (9m) 43s ago 31m 87.3M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 2m ago 30m 61.4M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (10m) 43s ago 35m 452M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (20m) 2m ago 37m 507M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (7m) 2m ago 37m 53.6M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (8m) 43s ago 35m 48.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (6m) 2m ago 36m 39.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (10m) 2m ago 31m 15.5M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (10m) 43s ago 31m 17.0M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (5m) 2m ago 35m 45.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (4m) 2m ago 35m 53.4M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:51:45.889 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (3m) 2m ago 34m 42.0M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (2m) 2m ago 34m 14.3M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (2m) 43s ago 33m 49.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (116s) 43s ago 33m 53.9M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (82s) 43s ago 33m 41.9M 3248M 19.0.0-1430-g8727ce93 170846cf6432 f5aa07b95e17 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (48s) 43s ago 32m 12.6M 3248M 19.0.0-1430-g8727ce93 170846cf6432 3685781bd0e6 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (9m) 43s ago 32m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (30m) 2m ago 30m 80.3M - 17.2.0 e1d6a67b021e 7649266d4d79 2024-02-14T17:51:45.890 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (30m) 43s ago 30m 81.4M - 17.2.0 e1d6a67b021e 4821953f3bb0 2024-02-14T17:51:46.306 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:51:46.306 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 8 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 13 2024-02-14T17:51:46.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:51:46.308 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:51:46.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:46.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='client.54592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='client.44658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='client.54604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.591 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.592 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1226524071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='client.54592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='client.44658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='client.54604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.593 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.594 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.594 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.594 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:46 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1226524071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "0/2 daemons upgraded", 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading rgw daemons", 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:51:46.673 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:51:46.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:46.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='client.54592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='client.44658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='client.54604 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:46.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:46 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1226524071' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:51:47.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:51:47.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:51:47.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mgr 2024-02-14T17:51:47.624 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mon 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all crash 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all osd 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mds 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[83585]: from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mgr 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mon 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all crash 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all osd 2024-02-14T17:51:47.625 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mds 2024-02-14T17:51:47.626 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:47 smithi017 ceph-mon[88777]: from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:47.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:51:47.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:51:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mgr 2024-02-14T17:51:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mon 2024-02-14T17:51:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all crash 2024-02-14T17:51:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all osd 2024-02-14T17:51:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mds 2024-02-14T17:51:47.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:47 smithi111 ceph-mon[68099]: from='client.44676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[83585]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[83585]: Upgrade: Updating rgw.foo.smithi017.vbwecr (1/2) 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[83585]: Deploying daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[88777]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:48.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[88777]: Upgrade: Updating rgw.foo.smithi017.vbwecr (1/2) 2024-02-14T17:51:48.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:48.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:51:48.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:48.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:48 smithi017 ceph-mon[88777]: Deploying daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:51:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:48 smithi111 ceph-mon[68099]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:48 smithi111 ceph-mon[68099]: Upgrade: Updating rgw.foo.smithi017.vbwecr (1/2) 2024-02-14T17:51:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:48 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:48 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi017.vbwecr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:51:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:48 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:48.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:48 smithi111 ceph-mon[68099]: Deploying daemon rgw.foo.smithi017.vbwecr on smithi017 2024-02-14T17:51:50.636 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:50 smithi017 ceph-mon[83585]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 0 op/s 2024-02-14T17:51:50.637 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:50 smithi017 ceph-mon[88777]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 0 op/s 2024-02-14T17:51:50.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:50 smithi111 ceph-mon[68099]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 0 op/s 2024-02-14T17:51:51.776 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:51.776 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:51.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:51:52.189 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:51:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:51:51] "GET /metrics HTTP/1.1" 200 38377 "" "Prometheus/2.43.0" 2024-02-14T17:51:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:52 smithi017 ceph-mon[88777]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:52.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:52.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:52 smithi017 ceph-mon[83585]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:52.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:52.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:52.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:52 smithi111 ceph-mon[68099]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:51:52.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:52.984 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:54.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:54 smithi111 ceph-mon[68099]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 102 B/s wr, 37 op/s 2024-02-14T17:51:54.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:54 smithi111 ceph-mon[68099]: Upgrade: Updating rgw.foo.smithi111.zvlcwh (2/2) 2024-02-14T17:51:54.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:54 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:54.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:54 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:51:54.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:54 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:54.415 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:54 smithi111 ceph-mon[68099]: Deploying daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[88777]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 102 B/s wr, 37 op/s 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[88777]: Upgrade: Updating rgw.foo.smithi111.zvlcwh (2/2) 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[88777]: Deploying daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[83585]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 102 B/s wr, 37 op/s 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[83585]: Upgrade: Updating rgw.foo.smithi111.zvlcwh (2/2) 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi111.zvlcwh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-14T17:51:54.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:51:54.691 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:54 smithi017 ceph-mon[83585]: Deploying daemon rgw.foo.smithi111.zvlcwh on smithi111 2024-02-14T17:51:56.414 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:56 smithi111 ceph-mon[68099]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 85 B/s wr, 68 op/s 2024-02-14T17:51:56.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:56 smithi017 ceph-mon[83585]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 85 B/s wr, 68 op/s 2024-02-14T17:51:56.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:56 smithi017 ceph-mon[88777]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 85 B/s wr, 68 op/s 2024-02-14T17:51:58.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[83585]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[88777]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-14T17:51:58.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:51:58 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:51:58.203 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:58.203 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:51:58.203 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:58 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:51:58.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:58 smithi111 ceph-mon[68099]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-02-14T17:51:58.204 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:51:58 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:00.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:00 smithi111 ceph-mon[68099]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 85 B/s wr, 124 op/s 2024-02-14T17:52:00.598 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:00 smithi017 ceph-mon[83585]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 85 B/s wr, 124 op/s 2024-02-14T17:52:00.598 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:00 smithi017 ceph-mon[88777]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 85 B/s wr, 124 op/s 2024-02-14T17:52:02.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:52:01] "GET /metrics HTTP/1.1" 200 38377 "" "Prometheus/2.43.0" 2024-02-14T17:52:02.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:01 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:01 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:01 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:01 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:02.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:01 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[83585]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 85 B/s wr, 139 op/s 2024-02-14T17:52:03.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.180 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.181 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.181 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[88777]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 85 B/s wr, 139 op/s 2024-02-14T17:52:03.181 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.181 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.181 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.181 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:02 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:02 smithi111 ceph-mon[68099]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 85 B/s wr, 139 op/s 2024-02-14T17:52:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:02 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:04 smithi017 ceph-mon[83585]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-14T17:52:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:04 smithi017 ceph-mon[88777]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-14T17:52:05.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:04 smithi111 ceph-mon[68099]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-02-14T17:52:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 85 B/s wr, 138 op/s 2024-02-14T17:52:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 99 B/s wr, 118 op/s 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 124 B/s wr, 123 op/s 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all rgw 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.vbwecr"}]: dispatch 2024-02-14T17:52:06.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.vbwecr"}]': finished 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi111.zvlcwh"}]: dispatch 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi111.zvlcwh"}]': finished 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.442 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 85 B/s wr, 138 op/s 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 99 B/s wr, 118 op/s 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 124 B/s wr, 123 op/s 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.443 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all rgw 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.vbwecr"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.vbwecr"}]': finished 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi111.zvlcwh"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi111.zvlcwh"}]': finished 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.444 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nfs 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Upgrade: Finalizing container_image settings 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:52:06.445 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-14T17:52:06.446 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nfs 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.447 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Upgrade: Finalizing container_image settings 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:52:06.448 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:52:06.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:52:06.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-14T17:52:06.449 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:52:06.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:52:06.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:52:06.449 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Upgrade: Complete! 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:52:06.450 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 157 B/s wr, 82 op/s 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 228 B/s wr, 80 op/s 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:52:06.451 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Upgrade: Complete! 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 157 B/s wr, 82 op/s 2024-02-14T17:52:06.452 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 228 B/s wr, 80 op/s 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: Checking dashboard <-> RGW credentials 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: Checking dashboard <-> RGW credentials 2024-02-14T17:52:06.453 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.454 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 85 B/s wr, 138 op/s 2024-02-14T17:52:06.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 99 B/s wr, 118 op/s 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 124 B/s wr, 123 op/s 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all rgw 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.485 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.vbwecr"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi017.vbwecr"}]': finished 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi111.zvlcwh"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi111.zvlcwh"}]': finished 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nfs 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all nvmeof 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.486 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Upgrade: Finalizing container_image settings 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-02-14T17:52:06.487 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.488 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Upgrade: Complete! 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 157 B/s wr, 82 op/s 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 53 KiB/s rd, 228 B/s wr, 80 op/s 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: Checking dashboard <-> RGW credentials 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:06.489 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:07.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:07 smithi017 ceph-mon[88777]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-14T17:52:07.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:07 smithi017 ceph-mon[88777]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:52:07.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:07 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:07 smithi017 ceph-mon[83585]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-14T17:52:07.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:07 smithi017 ceph-mon[83585]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:52:07.441 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:07 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:07.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:07 smithi111 ceph-mon[68099]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-02-14T17:52:07.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:07 smithi111 ceph-mon[68099]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:52:07.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:07 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:08 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:08 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:08.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:08 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:09.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:09 smithi017 ceph-mon[83585]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 6 op/s 2024-02-14T17:52:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:09 smithi017 ceph-mon[88777]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 6 op/s 2024-02-14T17:52:09.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:09 smithi111 ceph-mon[68099]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 6 op/s 2024-02-14T17:52:11.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:11 smithi017 ceph-mon[83585]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 6 op/s 2024-02-14T17:52:11.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:11 smithi017 ceph-mon[88777]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 6 op/s 2024-02-14T17:52:11.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:11 smithi111 ceph-mon[68099]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 6 op/s 2024-02-14T17:52:12.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:52:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:52:11] "GET /metrics HTTP/1.1" 200 38410 "" "Prometheus/2.43.0" 2024-02-14T17:52:13.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:13 smithi017 ceph-mon[83585]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-02-14T17:52:13.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:13 smithi017 ceph-mon[88777]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-02-14T17:52:13.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:13 smithi111 ceph-mon[68099]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 4 op/s 2024-02-14T17:52:15.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:15 smithi017 ceph-mon[83585]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-02-14T17:52:15.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:15 smithi017 ceph-mon[88777]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-02-14T17:52:15.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:15 smithi111 ceph-mon[68099]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s 2024-02-14T17:52:17.337 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:17 smithi017 ceph-mon[83585]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-14T17:52:17.338 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:17 smithi017 ceph-mon[88777]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-14T17:52:17.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:17 smithi111 ceph-mon[68099]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-02-14T17:52:17.759 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:52:18.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:18 smithi017 ceph-mon[83585]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:18.370 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:18 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:18.370 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:18 smithi017 ceph-mon[88777]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:18.370 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:18 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:18.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:18 smithi111 ceph-mon[68099]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:18.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:18 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:19.252 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:19 smithi017 ceph-mon[83585]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-14T17:52:19.252 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:19 smithi017 ceph-mon[88777]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-14T17:52:19.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:19 smithi111 ceph-mon[68099]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (10m) 18s ago 31m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (9m) 18s ago 31m 90.1M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 18s ago 31m 63.7M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (11m) 18s ago 36m 453M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (21m) 18s ago 37m 513M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (7m) 18s ago 38m 61.5M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (8m) 18s ago 36m 50.0M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (7m) 18s ago 36m 44.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:52:19.735 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (10m) 18s ago 32m 15.6M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (10m) 18s ago 32m 17.0M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (5m) 18s ago 36m 54.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (5m) 18s ago 35m 62.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (4m) 18s ago 35m 47.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (3m) 18s ago 34m 56.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (2m) 18s ago 34m 54.4M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (2m) 18s ago 34m 58.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (116s) 18s ago 33m 46.2M 3248M 19.0.0-1430-g8727ce93 170846cf6432 f5aa07b95e17 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (82s) 18s ago 33m 55.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 3685781bd0e6 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (10m) 18s ago 32m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (28s) 18s ago 31m 77.3M - 19.0.0-1430-g8727ce93 170846cf6432 3b0d4289ea40 2024-02-14T17:52:19.736 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (23s) 18s ago 31m 77.1M - 19.0.0-1430-g8727ce93 170846cf6432 5130bda08578 2024-02-14T17:52:20.322 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:20 smithi017 ceph-mon[83585]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:20.330 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:20 smithi017 ceph-mon[88777]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:20.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-02-14T17:52:20.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:20 smithi111 ceph-mon[68099]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:21 smithi017 ceph-mon[83585]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-02-14T17:52:21.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:21 smithi017 ceph-mon[88777]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-02-14T17:52:21.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:21.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:21 smithi111 ceph-mon[68099]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-02-14T17:52:21.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:21.873 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:52:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:52:21] "GET /metrics HTTP/1.1" 200 38403 "" "Prometheus/2.43.0" 2024-02-14T17:52:22.401 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:52:22.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-02-14T17:52:23.271 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:23 smithi017 ceph-mon[83585]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:23.271 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:23 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2473448197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:23.271 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:23 smithi017 ceph-mon[88777]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:23.271 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:23 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2473448197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:23.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:23 smithi111 ceph-mon[68099]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:23.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:23 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2473448197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:25.009 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)" 2024-02-14T17:52:25.290 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:25 smithi017 ceph-mon[83585]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:25.290 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:25 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2664101084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:25.290 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:25 smithi017 ceph-mon[88777]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:25.290 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:25 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2664101084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:25.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:25 smithi111 ceph-mon[68099]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:25.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:25 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2664101084' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:25.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:52:27.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:27 smithi017 ceph-mon[83585]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:27.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:27 smithi017 ceph-mon[88777]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:27.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:27 smithi111 ceph-mon[68099]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": null, 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": false, 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "", 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [], 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": null, 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "", 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:52:28.312 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:28 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:28.312 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:28 smithi017 ceph-mon[88777]: from='client.54730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:28.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:28 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:28.312 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:28 smithi017 ceph-mon[83585]: from='client.54730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:28.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:52:28.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:28 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:28.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:28 smithi111 ceph-mon[68099]: from='client.54730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:29.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:29 smithi111 ceph-mon[68099]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:29.484 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:52:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a[66485]: logger=cleanup t=2024-02-14T17:52:29.333778342Z level=info msg="Completed cleanup jobs" duration=3.228799ms 2024-02-14T17:52:29.636 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:29 smithi017 ceph-mon[83585]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:29.636 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:29 smithi017 ceph-mon[88777]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:30.414 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:52:30.414 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:52:30.414 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:52:31.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-02-14T17:52:31.363 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:31 smithi017 ceph-mon[88777]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:31.363 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:31 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1125335076' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:52:31.363 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:31 smithi017 ceph-mon[83585]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:31.363 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:31 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1125335076' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:52:31.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:31 smithi111 ceph-mon[68099]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:31.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:31 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1125335076' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:52:31.939 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:52:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:52:31] "GET /metrics HTTP/1.1" 200 38403 "" "Prometheus/2.43.0" 2024-02-14T17:52:32.956 INFO:teuthology.orchestra.run.smithi017.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:52:33.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:33 smithi111 ceph-mon[68099]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:33 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:33 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:33 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:33 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:33.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:33 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:33.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[88777]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:33.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:33.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:33.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:33.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:33.608 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:33.609 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[83585]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:33.609 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:33.609 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:33.609 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:33.609 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:33.609 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:33 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:33.635 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-14T17:52:33.647 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-02-14T17:52:33.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-02-14T17:52:34.230 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[88777]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[88777]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[88777]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[88777]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[88777]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[83585]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[83585]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[83585]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[83585]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:34.231 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:34 smithi017 ceph-mon[83585]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:52:34.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:34 smithi111 ceph-mon[68099]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:34 smithi111 ceph-mon[68099]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:52:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:34 smithi111 ceph-mon[68099]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 928 B/s rd, 0 op/s 2024-02-14T17:52:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:34 smithi111 ceph-mon[68099]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:34.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:34 smithi111 ceph-mon[68099]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:8727ce939fc044a98f215e29696bb1433f5758d1 2024-02-14T17:52:36.948 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:52:37.123 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:37.123 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.123 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:52:37.123 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:52:37.123 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:37.123 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.123 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mgr 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mon 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all crash 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all osd 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all mds 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.124 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all rgw 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:52:37.125 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mgr 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mon 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all crash 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all osd 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all mds 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.126 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all rgw 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.127 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Target is version 19.0.0-1430-g8727ce93 (squid) 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b'] 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mgr 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mon 2024-02-14T17:52:37.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all crash 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all osd 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all mds 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all rgw 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all rbd-mirror 2024-02-14T17:52:37.235 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-14T17:52:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: Upgrade: Setting container_image for all ceph-exporter 2024-02-14T17:52:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:37.236 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:37.619 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:52:37.619 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (10m) 36s ago 32m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:52:37.619 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (10m) 36s ago 31m 90.1M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 36s ago 31m 63.7M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (11m) 36s ago 36m 453M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (21m) 36s ago 38m 513M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (8m) 36s ago 38m 61.5M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (9m) 36s ago 36m 50.0M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (7m) 36s ago 37m 44.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (11m) 36s ago 32m 15.6M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (10m) 36s ago 32m 17.0M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (6m) 36s ago 36m 54.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (5m) 36s ago 35m 62.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (4m) 36s ago 35m 47.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (3m) 36s ago 35m 56.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (3m) 36s ago 34m 54.4M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (2m) 36s ago 34m 58.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:52:37.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (2m) 36s ago 34m 46.2M 3248M 19.0.0-1430-g8727ce93 170846cf6432 f5aa07b95e17 2024-02-14T17:52:37.621 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (100s) 36s ago 33m 55.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 3685781bd0e6 2024-02-14T17:52:37.621 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (10m) 36s ago 32m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:52:37.621 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (46s) 36s ago 31m 77.3M - 19.0.0-1430-g8727ce93 170846cf6432 3b0d4289ea40 2024-02-14T17:52:37.621 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (41s) 36s ago 31m 77.1M - 19.0.0-1430-g8727ce93 170846cf6432 5130bda08578 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:52:38.031 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 8 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 15 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:52:38.032 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:52:38.374 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:52:38.374 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "osd", 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw", 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "15/21 daemons upgraded", 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Currently upgrading iscsi daemons", 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": false 2024-02-14T17:52:38.375 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:52:38.795 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-02-14T17:52:38.795 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:52:38.795 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='client.44805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='client.54754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='client.44817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:39.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/3449405348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/3230360536' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='client.44805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='client.54754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='client.44817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/3449405348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:39.191 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:38 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/3230360536' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:52:39.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='client.44805 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:39.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='client.54754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:39.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='client.44817 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi017.wtnmjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-14T17:52:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/3449405348' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:52:39.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:38 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/3230360536' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:52:39.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:39 smithi017 ceph-mon[83585]: Upgrade: Updating iscsi.foo.smithi017.wtnmjz 2024-02-14T17:52:39.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:39 smithi017 ceph-mon[83585]: Deploying daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:52:39.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:39 smithi017 ceph-mon[83585]: from='client.44829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:39 smithi017 ceph-mon[88777]: Upgrade: Updating iscsi.foo.smithi017.wtnmjz 2024-02-14T17:52:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:39 smithi017 ceph-mon[88777]: Deploying daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:52:39.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:39 smithi017 ceph-mon[88777]: from='client.44829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:40.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:39 smithi111 ceph-mon[68099]: Upgrade: Updating iscsi.foo.smithi017.wtnmjz 2024-02-14T17:52:40.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:39 smithi111 ceph-mon[68099]: Deploying daemon iscsi.foo.smithi017.wtnmjz on smithi017 2024-02-14T17:52:40.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:39 smithi111 ceph-mon[68099]: from='client.44829 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:52:41.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:40 smithi017 ceph-mon[83585]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-02-14T17:52:41.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:40 smithi017 ceph-mon[88777]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-02-14T17:52:41.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:40 smithi111 ceph-mon[68099]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 567 B/s rd, 0 op/s 2024-02-14T17:52:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:52:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:52:41] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-02-14T17:52:43.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:42 smithi017 ceph-mon[83585]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:43.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:42 smithi017 ceph-mon[88777]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:43.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:42 smithi111 ceph-mon[68099]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:52:45.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:44 smithi017 ceph-mon[83585]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:52:45.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:44 smithi017 ceph-mon[88777]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:52:45.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:44 smithi111 ceph-mon[68099]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:52:47.089 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:46 smithi017 ceph-mon[83585]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:47.089 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:46 smithi017 ceph-mon[88777]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:47.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:46 smithi111 ceph-mon[68099]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:49.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:48 smithi017 ceph-mon[83585]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:49.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:48 smithi017 ceph-mon[83585]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:49.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:48 smithi017 ceph-mon[88777]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:49.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:48 smithi017 ceph-mon[88777]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:49.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:48 smithi111 ceph-mon[68099]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:49.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:48 smithi111 ceph-mon[68099]: from='client.25258 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:52:51.109 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:50 smithi017 ceph-mon[83585]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:51.110 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:50 smithi017 ceph-mon[88777]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:51.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:50 smithi111 ceph-mon[68099]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:52:51.398 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:52:51.107+0000 7f1c71dfb700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. 2024-02-14T17:52:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:52.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:52.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:52:51] "GET /metrics HTTP/1.1" 200 38410 "" "Prometheus/2.43.0" 2024-02-14T17:52:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:52.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:52.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:52:52.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:52.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:52:52.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:52:52.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:52:52.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[83585]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[83585]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[83585]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[83585]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[83585]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[83585]: Health check failed: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[88777]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[88777]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. 2024-02-14T17:52:53.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[88777]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s 2024-02-14T17:52:53.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[88777]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:53.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[88777]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-02-14T17:52:53.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:52 smithi017 ceph-mon[88777]: Health check failed: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-14T17:52:53.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:52 smithi111 ceph-mon[68099]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:53.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:52 smithi111 ceph-mon[68099]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. 2024-02-14T17:52:53.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:52 smithi111 ceph-mon[68099]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s 2024-02-14T17:52:53.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:52 smithi111 ceph-mon[68099]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:52:53.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:52 smithi111 ceph-mon[68099]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-02-14T17:52:53.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:52 smithi111 ceph-mon[68099]: Health check failed: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-02-14T17:52:55.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:54 smithi017 ceph-mon[83585]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-02-14T17:52:55.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:54 smithi017 ceph-mon[88777]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-02-14T17:52:55.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:54 smithi111 ceph-mon[68099]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-02-14T17:52:57.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:56 smithi017 ceph-mon[83585]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-02-14T17:52:57.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:56 smithi017 ceph-mon[88777]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-02-14T17:52:57.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:56 smithi111 ceph-mon[68099]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-02-14T17:52:59.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:52:58 smithi017 ceph-mon[83585]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-14T17:52:59.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:52:58 smithi017 ceph-mon[88777]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-14T17:52:59.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:52:58 smithi111 ceph-mon[68099]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 167 B/s rd, 0 op/s 2024-02-14T17:53:01.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:00 smithi017 ceph-mon[83585]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-02-14T17:53:01.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:00 smithi017 ceph-mon[88777]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-02-14T17:53:01.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:00 smithi111 ceph-mon[68099]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-02-14T17:53:01.960 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:53:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:53:01] "GET /metrics HTTP/1.1" 200 38410 "" "Prometheus/2.43.0" 2024-02-14T17:53:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[83585]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-14T17:53:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/395241568' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:53:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2384363479' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]: dispatch 2024-02-14T17:53:03.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[83585]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]: dispatch 2024-02-14T17:53:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[88777]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-14T17:53:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/395241568' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:53:03.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2384363479' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]: dispatch 2024-02-14T17:53:03.191 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:02 smithi017 ceph-mon[88777]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]: dispatch 2024-02-14T17:53:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:02 smithi111 ceph-mon[68099]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-14T17:53:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:02 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/395241568' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-14T17:53:03.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:02 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2384363479' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]: dispatch 2024-02-14T17:53:03.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:02 smithi111 ceph-mon[68099]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]: dispatch 2024-02-14T17:53:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:03 smithi017 ceph-mon[83585]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]': finished 2024-02-14T17:53:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:03 smithi017 ceph-mon[83585]: osdmap e153: 8 total, 8 up, 8 in 2024-02-14T17:53:04.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:03 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2204281089' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2926644197"}]: dispatch 2024-02-14T17:53:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:03 smithi017 ceph-mon[88777]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]': finished 2024-02-14T17:53:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:03 smithi017 ceph-mon[88777]: osdmap e153: 8 total, 8 up, 8 in 2024-02-14T17:53:04.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:03 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2204281089' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2926644197"}]: dispatch 2024-02-14T17:53:04.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:03 smithi111 ceph-mon[68099]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/4155023103"}]': finished 2024-02-14T17:53:04.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:03 smithi111 ceph-mon[68099]: osdmap e153: 8 total, 8 up, 8 in 2024-02-14T17:53:04.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:03 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2204281089' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2926644197"}]: dispatch 2024-02-14T17:53:05.189 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[83585]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2204281089' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2926644197"}]': finished 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[83585]: osdmap e154: 8 total, 8 up, 8 in 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/7408530' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]: dispatch 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[83585]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]: dispatch 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[88777]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2204281089' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2926644197"}]': finished 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[88777]: osdmap e154: 8 total, 8 up, 8 in 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/7408530' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]: dispatch 2024-02-14T17:53:05.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:04 smithi017 ceph-mon[88777]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]: dispatch 2024-02-14T17:53:05.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:04 smithi111 ceph-mon[68099]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-02-14T17:53:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:04 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2204281089' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/2926644197"}]': finished 2024-02-14T17:53:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:04 smithi111 ceph-mon[68099]: osdmap e154: 8 total, 8 up, 8 in 2024-02-14T17:53:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:04 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/7408530' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]: dispatch 2024-02-14T17:53:05.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:04 smithi111 ceph-mon[68099]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]: dispatch 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[88777]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]': finished 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[88777]: osdmap e155: 8 total, 8 up, 8 in 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/3231533016' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]: dispatch 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[88777]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]: dispatch 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[83585]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]': finished 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[83585]: osdmap e155: 8 total, 8 up, 8 in 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/3231533016' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]: dispatch 2024-02-14T17:53:06.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:05 smithi017 ceph-mon[83585]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]: dispatch 2024-02-14T17:53:06.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:05 smithi111 ceph-mon[68099]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/3269366497"}]': finished 2024-02-14T17:53:06.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:05 smithi111 ceph-mon[68099]: osdmap e155: 8 total, 8 up, 8 in 2024-02-14T17:53:06.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:05 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/3231533016' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]: dispatch 2024-02-14T17:53:06.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:05 smithi111 ceph-mon[68099]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]: dispatch 2024-02-14T17:53:07.189 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[88777]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[88777]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]': finished 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[88777]: osdmap e156: 8 total, 8 up, 8 in 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/26534905' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2630782886"}]: dispatch 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[83585]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[83585]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]': finished 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[83585]: osdmap e156: 8 total, 8 up, 8 in 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:07.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:06 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/26534905' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2630782886"}]: dispatch 2024-02-14T17:53:07.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:06 smithi111 ceph-mon[68099]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-14T17:53:07.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:06 smithi111 ceph-mon[68099]: from='client.? ' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:0/1841156916"}]': finished 2024-02-14T17:53:07.233 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:06 smithi111 ceph-mon[68099]: osdmap e156: 8 total, 8 up, 8 in 2024-02-14T17:53:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:53:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:07.234 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:06 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/26534905' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2630782886"}]: dispatch 2024-02-14T17:53:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/26534905' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2630782886"}]': finished 2024-02-14T17:53:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[83585]: osdmap e157: 8 total, 8 up, 8 in 2024-02-14T17:53:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[83585]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:53:08.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2302211940' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2630782886"}]: dispatch 2024-02-14T17:53:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/26534905' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2630782886"}]': finished 2024-02-14T17:53:08.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[88777]: osdmap e157: 8 total, 8 up, 8 in 2024-02-14T17:53:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[88777]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:53:08.441 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:08 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2302211940' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2630782886"}]: dispatch 2024-02-14T17:53:08.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:08 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/26534905' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6800/2630782886"}]': finished 2024-02-14T17:53:08.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:08 smithi111 ceph-mon[68099]: osdmap e157: 8 total, 8 up, 8 in 2024-02-14T17:53:08.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:08 smithi111 ceph-mon[68099]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-14T17:53:08.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:08 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2302211940' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2630782886"}]: dispatch 2024-02-14T17:53:09.168 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:53:09.439 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:09 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2302211940' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2630782886"}]': finished 2024-02-14T17:53:09.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:09 smithi017 ceph-mon[83585]: osdmap e158: 8 total, 8 up, 8 in 2024-02-14T17:53:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:09 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2302211940' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2630782886"}]': finished 2024-02-14T17:53:09.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:09 smithi017 ceph-mon[88777]: osdmap e158: 8 total, 8 up, 8 in 2024-02-14T17:53:09.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:09 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2302211940' entity='client.iscsi.foo.smithi017.wtnmjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.17:6801/2630782886"}]': finished 2024-02-14T17:53:09.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:09 smithi111 ceph-mon[68099]: osdmap e158: 8 total, 8 up, 8 in 2024-02-14T17:53:09.503 INFO:teuthology.orchestra.run.smithi017.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed." 2024-02-14T17:53:10.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:53:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:10 smithi017 ceph-mon[83585]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:53:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:10 smithi017 ceph-mon[83585]: from='client.54844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:10.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:10 smithi017 ceph-mon[83585]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:10 smithi017 ceph-mon[88777]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:53:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:10 smithi017 ceph-mon[88777]: from='client.54844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:10.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:10 smithi017 ceph-mon[88777]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:10.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:10 smithi111 ceph-mon[68099]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail 2024-02-14T17:53:10.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:10 smithi111 ceph-mon[68099]: from='client.54844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:10.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:10 smithi111 ceph-mon[68099]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:11.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:53:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:53:11] "GET /metrics HTTP/1.1" 200 38488 "" "Prometheus/2.43.0" 2024-02-14T17:53:12.176 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:53:12.176 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (11m) 71s ago 32m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:53:12.176 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (10m) 71s ago 32m 90.1M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:53:12.176 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 71s ago 31m 63.7M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:53:12.176 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (11m) 71s ago 37m 453M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:53:12.176 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (22m) 71s ago 38m 513M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (8m) 71s ago 39m 61.5M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (9m) 71s ago 37m 50.0M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (8m) 71s ago 37m 44.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (11m) 71s ago 33m 15.6M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (11m) 71s ago 33m 17.0M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (6m) 71s ago 36m 54.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (6m) 71s ago 36m 62.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (5m) 71s ago 36m 47.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (4m) 71s ago 35m 56.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (3m) 71s ago 35m 54.4M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (3m) 71s ago 35m 58.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (2m) 71s ago 34m 46.2M 3248M 19.0.0-1430-g8727ce93 170846cf6432 f5aa07b95e17 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (2m) 71s ago 34m 55.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 3685781bd0e6 2024-02-14T17:53:12.177 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (11m) 71s ago 33m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:53:12.178 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (80s) 71s ago 32m 77.3M - 19.0.0-1430-g8727ce93 170846cf6432 3b0d4289ea40 2024-02-14T17:53:12.178 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (75s) 71s ago 32m 77.1M - 19.0.0-1430-g8727ce93 170846cf6432 5130bda08578 2024-02-14T17:53:12.190 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:12 smithi017 ceph-mon[83585]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:12.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:12 smithi017 ceph-mon[88777]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:12.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:12 smithi111 ceph-mon[68099]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:12.772 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions' 2024-02-14T17:53:13.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:13 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:13.376 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:13 smithi017 ceph-mon[83585]: from='client.54856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:13.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:13 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:13.377 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:13 smithi017 ceph-mon[88777]: from='client.54856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:13.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:13 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:13.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:13 smithi111 ceph-mon[68099]: from='client.54856 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:53:14.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:14 smithi017 ceph-mon[83585]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-02-14T17:53:14.440 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:14 smithi017 ceph-mon[88777]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-02-14T17:53:14.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:14 smithi111 ceph-mon[68099]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 0 op/s 2024-02-14T17:53:14.609 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 8 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:53:14.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:53:14.611 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:53:14.611 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 15 2024-02-14T17:53:14.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:53:14.611 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:53:15.390 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:15 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2758487973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:53:15.390 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:15 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2758487973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:53:15.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-02-14T17:53:15.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:15 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2758487973' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:53:16.401 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:16 smithi017 ceph-mon[83585]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:16.401 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:16 smithi017 ceph-mon[88777]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:16.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:16 smithi111 ceph-mon[68099]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:17.129 INFO:teuthology.orchestra.run.smithi017.stdout:wait for servicemap items w/ changing names to refresh 2024-02-14T17:53:17.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'sleep 60' 2024-02-14T17:53:18.294 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:18 smithi017 ceph-mon[83585]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:53:18.295 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:18 smithi017 ceph-mon[88777]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:53:18.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:18 smithi111 ceph-mon[68099]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:53:20.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:20 smithi111 ceph-mon[68099]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-02-14T17:53:20.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:20 smithi017 ceph-mon[88777]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-02-14T17:53:20.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:20 smithi017 ceph-mon[83585]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-02-14T17:53:21.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:21.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:21.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:22.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:53:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:53:21] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-02-14T17:53:22.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:22 smithi111 ceph-mon[68099]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:22.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:22 smithi017 ceph-mon[83585]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:22.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:22 smithi017 ceph-mon[88777]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:23.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:23 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:23.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:23 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:23.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:23 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:24.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:24 smithi111 ceph-mon[68099]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:24.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:24 smithi017 ceph-mon[83585]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:24.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:24 smithi017 ceph-mon[88777]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:26.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:26 smithi111 ceph-mon[68099]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:26.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:26 smithi017 ceph-mon[83585]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:26.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:26 smithi017 ceph-mon[88777]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:28.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:28 smithi111 ceph-mon[68099]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:28 smithi017 ceph-mon[83585]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:28 smithi017 ceph-mon[88777]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:30.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:30 smithi111 ceph-mon[68099]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:30.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:30 smithi017 ceph-mon[83585]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:30.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:30 smithi017 ceph-mon[88777]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:32.189 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:53:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:53:31] "GET /metrics HTTP/1.1" 200 38486 "" "Prometheus/2.43.0" 2024-02-14T17:53:32.460 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:32 smithi017 ceph-mon[83585]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:32.460 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:32 smithi017 ceph-mon[88777]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:32.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:32 smithi111 ceph-mon[68099]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:33.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:33 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:33.689 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:33 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:33.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:33 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:34.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:34 smithi111 ceph-mon[68099]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:34.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:34 smithi017 ceph-mon[83585]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:34.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:34 smithi017 ceph-mon[88777]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:36 smithi111 ceph-mon[68099]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:36.484 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:36 smithi017 ceph-mon[88777]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:36 smithi017 ceph-mon[83585]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:38.483 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:38 smithi111 ceph-mon[68099]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:38.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:38 smithi017 ceph-mon[83585]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:38.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:38 smithi017 ceph-mon[88777]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:40.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:40 smithi017 ceph-mon[83585]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:40.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:40 smithi017 ceph-mon[88777]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:40.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:40 smithi111 ceph-mon[68099]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:42.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:53:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:53:41] "GET /metrics HTTP/1.1" 200 38482 "" "Prometheus/2.43.0" 2024-02-14T17:53:42.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:42 smithi017 ceph-mon[83585]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:42.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:42 smithi017 ceph-mon[88777]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:42.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:42 smithi111 ceph-mon[68099]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:43.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:43 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:43.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:43 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:43.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:43 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:44.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:44 smithi017 ceph-mon[83585]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:44.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:44 smithi017 ceph-mon[88777]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:44.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:44 smithi111 ceph-mon[68099]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:46.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:46 smithi017 ceph-mon[83585]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:46.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:46 smithi017 ceph-mon[88777]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:46.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:46 smithi111 ceph-mon[68099]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:48.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:48 smithi017 ceph-mon[83585]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:48.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:48 smithi017 ceph-mon[88777]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:48.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:48 smithi111 ceph-mon[68099]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:50.536 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:50 smithi111 ceph-mon[68099]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:50.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:50 smithi017 ceph-mon[83585]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:50.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:50 smithi017 ceph-mon[88777]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:53:51.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:51 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:51.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:51 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:51.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:51 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:53:52.192 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:53:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:53:51] "GET /metrics HTTP/1.1" 200 38488 "" "Prometheus/2.43.0" 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[83585]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[88777]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:52.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:53:52.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:53:52.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:52 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:53:52.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-14T17:53:52.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:52 smithi111 ceph-mon[68099]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:53:52.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-14T17:53:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-14T17:53:52.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:52 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' 2024-02-14T17:53:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:53 smithi017 ceph-mon[83585]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-02-14T17:53:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:53 smithi017 ceph-mon[83585]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:53.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:53 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:53.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:53 smithi017 ceph-mon[88777]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-02-14T17:53:53.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:53 smithi017 ceph-mon[88777]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:53.691 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:53 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:53.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:53 smithi111 ceph-mon[68099]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-02-14T17:53:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:53 smithi111 ceph-mon[68099]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:53.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:53 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:53:55.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:55 smithi017 ceph-mon[83585]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-02-14T17:53:55.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:55 smithi017 ceph-mon[88777]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-02-14T17:53:55.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:55 smithi111 ceph-mon[68099]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-02-14T17:53:57.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:57 smithi017 ceph-mon[83585]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:57.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:57 smithi017 ceph-mon[88777]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:57.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:57 smithi111 ceph-mon[68099]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:59.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:53:59 smithi017 ceph-mon[83585]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:59.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:53:59 smithi017 ceph-mon[88777]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:53:59.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:53:59 smithi111 ceph-mon[68099]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-14T17:54:01.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:01 smithi017 ceph-mon[83585]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:01.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:01 smithi017 ceph-mon[88777]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:01.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:01 smithi111 ceph-mon[68099]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:02.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:54:01] "GET /metrics HTTP/1.1" 200 38488 "" "Prometheus/2.43.0" 2024-02-14T17:54:03.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:03 smithi017 ceph-mon[83585]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:54:03.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:03 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:03 smithi017 ceph-mon[88777]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:54:03.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:03 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:03.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:03 smithi111 ceph-mon[68099]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-14T17:54:03.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:03 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:05.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:05 smithi017 ceph-mon[83585]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:05.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:05 smithi017 ceph-mon[88777]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:05.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:05 smithi111 ceph-mon[68099]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:06.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:06 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:06.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:06 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:06.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:06 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:07.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:07 smithi017 ceph-mon[83585]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:07.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:07 smithi017 ceph-mon[88777]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:07.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:07 smithi111 ceph-mon[68099]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:09.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:09 smithi017 ceph-mon[83585]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:09.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:09 smithi017 ceph-mon[88777]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:09.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:09 smithi111 ceph-mon[68099]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:11.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:11 smithi017 ceph-mon[83585]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:11.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:11 smithi017 ceph-mon[88777]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:11.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:11 smithi111 ceph-mon[68099]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:12.030 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:11 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:54:11] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.43.0" 2024-02-14T17:54:13.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:13 smithi017 ceph-mon[83585]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:13.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:13 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:13 smithi017 ceph-mon[88777]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:13.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:13 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:13.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:13 smithi111 ceph-mon[68099]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:13.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:13 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:15.689 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:15 smithi017 ceph-mon[83585]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:15.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:15 smithi017 ceph-mon[88777]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:15.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:15 smithi111 ceph-mon[68099]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:17.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:17 smithi017 ceph-mon[83585]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:17.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:17 smithi017 ceph-mon[88777]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:17.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:17 smithi111 ceph-mon[68099]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:19.602 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:19 smithi017 ceph-mon[83585]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:19.602 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:19 smithi017 ceph-mon[88777]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:19.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:19 smithi111 ceph-mon[68099]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:19.941 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ps' 2024-02-14T17:54:21.495 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:21 smithi017 ceph-mon[83585]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:21.496 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:21 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:21.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:21 smithi017 ceph-mon[88777]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:21.496 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:21 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:21.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:21 smithi111 ceph-mon[68099]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:21.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:21 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:21.944 INFO:teuthology.orchestra.run.smithi017.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-14T17:54:21.944 INFO:teuthology.orchestra.run.smithi017.stdout:alertmanager.a smithi017 *:9093,9094 running (12m) 2m ago 34m 25.6M - 0.25.0 c8568f914cd2 3ba623745a7f 2024-02-14T17:54:21.944 INFO:teuthology.orchestra.run.smithi017.stdout:grafana.a smithi111 *:3000 running (11m) 2m ago 33m 90.1M - 9.4.12 a0ea5e3c46d9 7b859cc99d08 2024-02-14T17:54:21.944 INFO:teuthology.orchestra.run.smithi017.stdout:iscsi.foo.smithi017.wtnmjz smithi017 error 2m ago 33m 63.7M - 3.5 e1d6a67b021e abfa5ed8cc93 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.x smithi111 *:8443,9283,8765 running (13m) 2m ago 38m 453M - 19.0.0-1430-g8727ce93 170846cf6432 8d2779e92e1b 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:mgr.y smithi017 *:8443,9283,8765 running (23m) 2m ago 39m 513M - 19.0.0-1430-g8727ce93 170846cf6432 bece6d782a35 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a smithi017 running (9m) 2m ago 40m 61.5M 2048M 19.0.0-1430-g8727ce93 170846cf6432 c0b8c4fc87ac 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b smithi111 running (10m) 2m ago 38m 50.0M 2048M 19.0.0-1430-g8727ce93 170846cf6432 bb055293bff4 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c smithi017 running (9m) 2m ago 38m 44.7M 2048M 19.0.0-1430-g8727ce93 170846cf6432 6ea55a625d8b 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.a smithi017 *:9100 running (12m) 2m ago 34m 15.6M - 1.5.0 0da6a335fe13 3ed6d4a4c5ad 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:node-exporter.b smithi111 *:9100 running (12m) 2m ago 34m 17.0M - 1.5.0 0da6a335fe13 b2920c9965f1 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 smithi017 running (7m) 2m ago 38m 54.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 15f522343f50 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 smithi017 running (7m) 2m ago 37m 62.5M 2993M 19.0.0-1430-g8727ce93 170846cf6432 8beec53ad366 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 smithi017 running (6m) 2m ago 37m 47.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 89cf7ada9e89 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 smithi017 running (5m) 2m ago 36m 56.7M 2993M 19.0.0-1430-g8727ce93 170846cf6432 f3a8d5491fd3 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 smithi111 running (4m) 2m ago 36m 54.4M 3248M 19.0.0-1430-g8727ce93 170846cf6432 c579b5723b70 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 smithi111 running (4m) 2m ago 36m 58.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 99350ba8e2aa 2024-02-14T17:54:21.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.6 smithi111 running (3m) 2m ago 35m 46.2M 3248M 19.0.0-1430-g8727ce93 170846cf6432 f5aa07b95e17 2024-02-14T17:54:21.946 INFO:teuthology.orchestra.run.smithi017.stdout:osd.7 smithi111 running (3m) 2m ago 35m 55.5M 3248M 19.0.0-1430-g8727ce93 170846cf6432 3685781bd0e6 2024-02-14T17:54:21.946 INFO:teuthology.orchestra.run.smithi017.stdout:prometheus.a smithi111 *:9095 running (12m) 2m ago 34m 62.5M - 2.43.0 a07b618ecd1d 51b0c15c913d 2024-02-14T17:54:21.946 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi017.vbwecr smithi017 *:8000 running (2m) 2m ago 33m 77.3M - 19.0.0-1430-g8727ce93 170846cf6432 3b0d4289ea40 2024-02-14T17:54:21.946 INFO:teuthology.orchestra.run.smithi017.stdout:rgw.foo.smithi111.zvlcwh smithi111 *:8000 running (2m) 2m ago 33m 77.1M - 19.0.0-1430-g8727ce93 170846cf6432 5130bda08578 2024-02-14T17:54:22.040 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:54:21] "GET /metrics HTTP/1.1" 200 38484 "" "Prometheus/2.43.0" 2024-02-14T17:54:22.599 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions' 2024-02-14T17:54:23.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:23 smithi111 ceph-mon[68099]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:23.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:23 smithi111 ceph-mon[68099]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:23.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:23 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:23.807 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:23 smithi017 ceph-mon[88777]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:23.807 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:23 smithi017 ceph-mon[88777]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:23.808 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:23 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:23.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:23 smithi017 ceph-mon[83585]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:23.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:23 smithi017 ceph-mon[83585]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:23.808 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:23 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "mon": { 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 3 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": { 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": { 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 8 2024-02-14T17:54:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:54:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw": { 2024-02-14T17:54:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 2 2024-02-14T17:54:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-02-14T17:54:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "overall": { 2024-02-14T17:54:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)": 15 2024-02-14T17:54:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-02-14T17:54:24.662 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:54:25.230 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch upgrade status' 2024-02-14T17:54:25.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:25 smithi017 ceph-mon[83585]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:25.544 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:25 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2362563666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:25.545 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:25 smithi017 ceph-mon[88777]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:25.545 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:25 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2362563666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:25.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:25 smithi111 ceph-mon[68099]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:25.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:25 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2362563666' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b", 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "in_progress": true, 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "which": "Upgrading all daemon types on all hosts", 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "services_complete": [ 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "osd", 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw", 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr", 2024-02-14T17:54:27.150 INFO:teuthology.orchestra.run.smithi017.stdout: "mon" 2024-02-14T17:54:27.151 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-02-14T17:54:27.151 INFO:teuthology.orchestra.run.smithi017.stdout: "progress": "15/21 daemons upgraded", 2024-02-14T17:54:27.151 INFO:teuthology.orchestra.run.smithi017.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed.", 2024-02-14T17:54:27.151 INFO:teuthology.orchestra.run.smithi017.stdout: "is_paused": true 2024-02-14T17:54:27.151 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:54:27.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:27 smithi111 ceph-mon[68099]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:27.744 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:27 smithi017 ceph-mon[83585]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:27.744 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:27 smithi017 ceph-mon[88777]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:27.769 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph health detail' 2024-02-14T17:54:28.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:28 smithi017 ceph-mon[83585]: from='client.44937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:28.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:28 smithi017 ceph-mon[88777]: from='client.44937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:28.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:28 smithi111 ceph-mon[68099]: from='client.44937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:29.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:29 smithi017 ceph-mon[83585]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:29.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:29 smithi017 ceph-mon[88777]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:29.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:29 smithi111 ceph-mon[68099]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:29.898 INFO:teuthology.orchestra.run.smithi017.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. 2024-02-14T17:54:29.898 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-02-14T17:54:29.898 INFO:teuthology.orchestra.run.smithi017.stdout: daemon iscsi.foo.smithi017.wtnmjz on smithi017 is in error state 2024-02-14T17:54:29.898 INFO:teuthology.orchestra.run.smithi017.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi017.wtnmjz on host smithi017 failed. 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrade daemon: iscsi.foo.smithi017.wtnmjz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi017.wtnmjz ... 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:Creating ceph-iscsi config... 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/iscsi-gateway.cfg 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:Write file: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/iscsi.foo.smithi017.wtnmjz/tcmu-runner-entrypoint.sh 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d612435b4\x2dcb5c\x2d11ee\x2d95b9\x2d87774f69a715.slice/ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:Non-zero exit code 1 from systemctl start ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:systemctl: stderr See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout:Traceback (most recent call last): 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout: return _run_code(code, main_globals, None, 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout: exec(code, run_globals) 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5520, in 2024-02-14T17:54:29.899 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 5508, in main 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3038, in command_deploy_from 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3073, in _common_deploy 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 3093, in _deploy_daemon_container 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1029, in deploy_daemon 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/__main__.py", line 1165, in deploy_daemon_units 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout: File "/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/cephadm.dbf8ae95bbaad79a9987ed057773c633546a843578278f6d6c60c48d0145903d/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout:RuntimeError: Failed command: systemctl start ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz: Job for ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service failed because the control process exited with error code. 2024-02-14T17:54:29.900 INFO:teuthology.orchestra.run.smithi017.stdout:See "systemctl status ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" and "journalctl -xeu ceph-612435b4-cb5c-11ee-95b9-87774f69a715@iscsi.foo.smithi017.wtnmjz.service" for details. 2024-02-14T17:54:30.550 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:30 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1687617680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:54:30.550 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:30 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1687617680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:54:30.574 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-02-14T17:54:30.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:30 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1687617680' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-02-14T17:54:31.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:31 smithi111 ceph-mon[68099]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:31.765 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:31 smithi017 ceph-mon[83585]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:31.765 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:31 smithi017 ceph-mon[88777]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:32.039 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:31 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:54:31] "GET /metrics HTTP/1.1" 200 38484 "" "Prometheus/2.43.0" 2024-02-14T17:54:32.567 INFO:teuthology.orchestra.run.smithi017.stdout:true 2024-02-14T17:54:33.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-02-14T17:54:33.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:33 smithi111 ceph-mon[68099]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:33 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:33.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:33 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/1450466547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:33.784 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:33 smithi017 ceph-mon[83585]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:33.784 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:33 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:33.784 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:33 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/1450466547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:33.784 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:33 smithi017 ceph-mon[88777]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:33.784 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:33 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:33.784 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:33 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/1450466547' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:35.175 INFO:teuthology.orchestra.run.smithi017.stdout: "ceph version 19.0.0-1430-g8727ce93 (8727ce939fc044a98f215e29696bb1433f5758d1) squid (dev)" 2024-02-14T17:54:35.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:35 smithi017 ceph-mon[83585]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:35.727 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:35 smithi017 ceph-mon[83585]: from='client.? 172.21.15.17:0/2181217958' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:35.728 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:35 smithi017 ceph-mon[88777]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:35.728 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:35 smithi017 ceph-mon[88777]: from='client.? 172.21.15.17:0/2181217958' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:35.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:35 smithi111 ceph-mon[68099]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:35.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:35 smithi111 ceph-mon[68099]: from='client.? 172.21.15.17:0/2181217958' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-14T17:54:35.752 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -e sha1=8727ce939fc044a98f215e29696bb1433f5758d1 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-14T17:54:36.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:36 smithi017 ceph-mon[88777]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:36.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:36 smithi017 ceph-mon[83585]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:36.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:36 smithi111 ceph-mon[68099]: from='mgr.25642 172.21.15.17:0/339897363' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-14T17:54:37.690 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:37 smithi017 ceph-mon[83585]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:37.690 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:37 smithi017 ceph-mon[88777]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:37.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:37 smithi111 ceph-mon[68099]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:37.764 INFO:teuthology.orchestra.run.smithi017.stdout:osd 8 2m ago - 2024-02-14T17:54:38.306 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-14T17:54:38.319 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi017.front.sepia.ceph.com 2024-02-14T17:54:38.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-02-14T17:54:38.631 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:38 smithi017 ceph-mon[83585]: from='client.44961 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:38.631 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:38 smithi017 ceph-mon[88777]: from='client.44961 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:38.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:38 smithi111 ceph-mon[68099]: from='client.44961 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:39.576 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:39 smithi017 ceph-mon[83585]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:39.576 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:39 smithi017 ceph-mon[88777]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:39.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:39 smithi111 ceph-mon[68099]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:40.981 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-02-14T17:54:40.981 INFO:teuthology.orchestra.run.smithi017.stdout: "image": "quay.io/ceph/ceph", 2024-02-14T17:54:40.982 INFO:teuthology.orchestra.run.smithi017.stdout: "registry": "quay.io", 2024-02-14T17:54:40.982 INFO:teuthology.orchestra.run.smithi017.stdout: "bare_image": "ceph/ceph", 2024-02-14T17:54:40.982 INFO:teuthology.orchestra.run.smithi017.stdout: "versions": [] 2024-02-14T17:54:40.982 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-02-14T17:54:41.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:41 smithi017 ceph-mon[83585]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:41.600 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:41 smithi017 ceph-mon[83585]: from='client.44967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:41.600 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:41 smithi017 ceph-mon[88777]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:41.601 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:41 smithi017 ceph-mon[88777]: from='client.44967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:41.602 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-02-14T17:54:41.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:41 smithi111 ceph-mon[68099]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:41.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:41 smithi111 ceph-mon[68099]: from='client.44967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:41.917 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:41 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:54:41] "GET /metrics HTTP/1.1" 200 38478 "" "Prometheus/2.43.0" 2024-02-14T17:54:43.939 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:43 smithi017 ceph-mon[88777]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:43.940 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:43 smithi017 ceph-mon[88777]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:43.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:43 smithi017 ceph-mon[83585]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:43.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:43 smithi017 ceph-mon[83585]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:43.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:43 smithi111 ceph-mon[68099]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:43.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:43 smithi111 ceph-mon[68099]: from='client.44850 -' entity='client.iscsi.foo.smithi017.wtnmjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-14T17:54:44.257 INFO:teuthology.orchestra.run.smithi017.stdout: "16.2.0", 2024-02-14T17:54:44.570 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:44 smithi017 ceph-mon[83585]: from='client.44973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:44.570 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:44 smithi017 ceph-mon[88777]: from='client.44973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:44.882 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-02-14T17:54:44.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:44 smithi111 ceph-mon[68099]: from='client.44973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:45.875 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:45 smithi017 ceph-mon[83585]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:45.875 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:45 smithi017 ceph-mon[88777]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:45.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:45 smithi111 ceph-mon[68099]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-14T17:54:47.502 INFO:teuthology.orchestra.run.smithi017.stdout: "v16.2.2", 2024-02-14T17:54:47.502 INFO:teuthology.orchestra.run.smithi017.stdout: "v16.2.2-20210505", 2024-02-14T17:54:47.774 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:47 smithi017 ceph-mon[83585]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:47.774 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:47 smithi017 ceph-mon[83585]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:47.774 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:47 smithi017 ceph-mon[88777]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:47.774 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:47 smithi017 ceph-mon[88777]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:47.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:47 smithi111 ceph-mon[68099]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-14T17:54:47.983 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:47 smithi111 ceph-mon[68099]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-14T17:54:48.066 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-14T17:54:48.075 INFO:tasks.cephadm:Teardown begin 2024-02-14T17:54:48.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:54:48.106 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:54:48.142 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-14T17:54:48.143 DEBUG:teuthology.orchestra.run.smithi017:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-14T17:54:48.162 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-14T17:54:48.199 INFO:tasks.cephadm:Stopping all daemons... 2024-02-14T17:54:48.199 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-14T17:54:48.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a 2024-02-14T17:54:48.440 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:48 smithi017 systemd[1]: Stopping Ceph mon.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:54:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a[83581]: 2024-02-14T17:54:48.526+0000 7fa34519b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:54:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a[83581]: 2024-02-14T17:54:48.526+0000 7fa34519b700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-02-14T17:54:48.940 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:48 smithi017 podman[130677]: 2024-02-14 17:54:48.690140438 +0000 UTC m=+0.295301462 container died c0b8c4fc87ac469fc6a314c1d295494a99631510bf1d1a46eda31bc750025d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:54:48.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:48] ENGINE Bus STOPPING 2024-02-14T17:54:48.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:54:48.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:48] ENGINE Bus STOPPED 2024-02-14T17:54:48.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:48] ENGINE Bus STARTING 2024-02-14T17:54:48.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: CherryPy Checker: 2024-02-14T17:54:48.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: The Application mounted at '' has an empty config. 2024-02-14T17:54:48.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:54:48.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:48] ENGINE Serving on http://:::9283 2024-02-14T17:54:48.941 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:48 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:48] ENGINE Bus STARTED 2024-02-14T17:54:49.286 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:49 smithi017 podman[130677]: 2024-02-14 17:54:49.076267919 +0000 UTC m=+0.681428989 container cleanup c0b8c4fc87ac469fc6a314c1d295494a99631510bf1d1a46eda31bc750025d7e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-14T17:54:49.286 INFO:journalctl@ceph.mon.a.smithi017.stdout:Feb 14 17:54:49 smithi017 bash[130677]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-a 2024-02-14T17:54:49.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.a.service' 2024-02-14T17:54:49.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:54:49.476 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-14T17:54:49.476 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-14T17:54:49.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.c 2024-02-14T17:54:49.823 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:49 smithi017 systemd[1]: Stopping Ceph mon.c for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:54:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c[88773]: 2024-02-14T17:54:49.821+0000 7f0ec72e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:54:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c[88773]: 2024-02-14T17:54:49.821+0000 7f0ec72e8700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-02-14T17:54:50.190 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:50 smithi017 podman[130809]: 2024-02-14 17:54:50.057578657 +0000 UTC m=+0.364868259 container died 6ea55a625d8bd76c6b19f250e2874fcd76f6b8ce8182e7c7711d2aee0f1a1121 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-14T17:54:50.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:49] ENGINE Bus STOPPING 2024-02-14T17:54:50.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-14T17:54:50.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:49] ENGINE Bus STOPPED 2024-02-14T17:54:50.190 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:49] ENGINE Bus STARTING 2024-02-14T17:54:50.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:49] ENGINE Serving on http://:::9283 2024-02-14T17:54:50.191 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:49 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: [14/Feb/2024:17:54:49] ENGINE Bus STARTED 2024-02-14T17:54:50.765 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:50 smithi017 podman[130809]: 2024-02-14 17:54:50.563888649 +0000 UTC m=+0.871178290 container cleanup 6ea55a625d8bd76c6b19f250e2874fcd76f6b8ce8182e7c7711d2aee0f1a1121 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=) 2024-02-14T17:54:50.765 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:50 smithi017 bash[130809]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c 2024-02-14T17:54:50.991 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.c.service' 2024-02-14T17:54:51.016 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:50 smithi017 podman[130834]: 2024-02-14 17:54:50.76483034 +0000 UTC m=+0.710853770 container remove 6ea55a625d8bd76c6b19f250e2874fcd76f6b8ce8182e7c7711d2aee0f1a1121 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:54:51.016 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:50 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-14T17:54:51.016 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:50 smithi017 systemd[1]: Stopped Ceph mon.c for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:54:51.016 INFO:journalctl@ceph.mon.c.smithi017.stdout:Feb 14 17:54:50 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.c.service: Consumed 6.117s CPU time. 2024-02-14T17:54:51.027 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:54:51.028 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-14T17:54:51.028 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-14T17:54:51.028 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.b 2024-02-14T17:54:51.340 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:51 smithi111 systemd[1]: Stopping Ceph mon.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:54:51.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b[68095]: 2024-02-14T17:54:51.339+0000 7fed29366700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:54:51.733 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b[68095]: 2024-02-14T17:54:51.339+0000 7fed29366700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-02-14T17:54:51.734 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:51 smithi111 podman[101530]: 2024-02-14 17:54:51.4721783 +0000 UTC m=+0.248541567 container died bb055293bff4f5e3fce301838f5eb64ce0f1016bc42d6ebbb61328fbe4065665 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-14T17:54:52.072 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:51 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: ::ffff:172.21.15.111 - - [14/Feb/2024:17:54:51] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.43.0" 2024-02-14T17:54:52.217 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:51 smithi111 podman[101530]: 2024-02-14 17:54:51.94028503 +0000 UTC m=+0.716648296 container cleanup bb055293bff4f5e3fce301838f5eb64ce0f1016bc42d6ebbb61328fbe4065665 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-14T17:54:52.217 INFO:journalctl@ceph.mon.b.smithi111.stdout:Feb 14 17:54:51 smithi111 bash[101530]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mon-b 2024-02-14T17:54:52.335 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mon.b.service' 2024-02-14T17:54:52.385 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:54:52.386 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-14T17:54:52.386 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-14T17:54:52.386 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y 2024-02-14T17:54:52.685 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:52 smithi017 systemd[1]: Stopping Ceph mgr.y for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:54:52.939 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:52 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:54:52.683+0000 7f1cdfe88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:54:52.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:52 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y[62957]: 2024-02-14T17:54:52.683+0000 7f1cdfe88700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-14T17:54:52.940 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:52 smithi017 podman[130942]: 2024-02-14 17:54:52.715221716 +0000 UTC m=+0.136045530 container died bece6d782a35ada8b7e774c6a6ed77384144475e8ed0083c461768073c81a08e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-14T17:54:53.328 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:53 smithi017 podman[130942]: 2024-02-14 17:54:53.101076282 +0000 UTC m=+0.521900102 container cleanup bece6d782a35ada8b7e774c6a6ed77384144475e8ed0083c461768073c81a08e (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y, ceph=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-14T17:54:53.328 INFO:journalctl@ceph.mgr.y.smithi017.stdout:Feb 14 17:54:53 smithi017 bash[130942]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-y 2024-02-14T17:54:53.472 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.y.service' 2024-02-14T17:54:53.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:54:53.510 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-14T17:54:53.510 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-14T17:54:53.510 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x 2024-02-14T17:54:53.852 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:54:53 smithi111 systemd[1]: Stopping Ceph mgr.x for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:54:54.138 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:54:53 smithi111 podman[101652]: 2024-02-14 17:54:53.851918122 +0000 UTC m=+0.151428107 container died 8d2779e92e1b144c8bef9bba76f9cad4a3e34a2b4efa1a7439b01c96e1ef0a30 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, org.label-schema.build-date=20240214, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:54:54.447 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:54:54 smithi111 podman[101652]: 2024-02-14 17:54:54.228982323 +0000 UTC m=+0.528492314 container cleanup 8d2779e92e1b144c8bef9bba76f9cad4a3e34a2b4efa1a7439b01c96e1ef0a30 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-14T17:54:54.447 INFO:journalctl@ceph.mgr.x.smithi111.stdout:Feb 14 17:54:54 smithi111 bash[101652]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-mgr-x 2024-02-14T17:54:54.587 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@mgr.x.service' 2024-02-14T17:54:54.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:54:54.625 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-14T17:54:54.625 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-14T17:54:54.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.0 2024-02-14T17:54:54.931 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:54:54 smithi017 systemd[1]: Stopping Ceph osd.0 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:54:55.190 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:54:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[96524]: 2024-02-14T17:54:54.929+0000 7f27bf472700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:54:55.190 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:54:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[96524]: 2024-02-14T17:54:54.929+0000 7f27bf472700 -1 osd.0 158 *** Got signal Terminated *** 2024-02-14T17:54:55.190 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:54:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0[96524]: 2024-02-14T17:54:54.929+0000 7f27bf472700 -1 osd.0 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:54:57.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:54:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:54:56.774Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:54:57.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:54:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:54:56.776Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:54:57.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:54:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:54:56.777Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:54:57.234 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:54:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:54:56.778Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:55:00.690 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:00 smithi017 podman[131062]: 2024-02-14 17:55:00.38215994 +0000 UTC m=+5.558945205 container died 15f522343f507c077320a56977b9c85f0120fddbf8e9842db9d34f1c0dc1e02f (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:55:01.190 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:00 smithi017 podman[131062]: 2024-02-14 17:55:00.713241313 +0000 UTC m=+5.890026578 container cleanup 15f522343f507c077320a56977b9c85f0120fddbf8e9842db9d34f1c0dc1e02f (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-14T17:55:01.190 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:00 smithi017 bash[131062]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0 2024-02-14T17:55:01.190 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:00 smithi017 podman[131079]: 2024-02-14 17:55:00.889264357 +0000 UTC m=+0.505236529 container remove 15f522343f507c077320a56977b9c85f0120fddbf8e9842db9d34f1c0dc1e02f (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:55:01.593 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:01 smithi017 podman[131174]: 2024-02-14 17:55:01.21555482 +0000 UTC m=+0.018969252 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:01.593 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:01 smithi017 podman[131174]: 2024-02-14 17:55:01.323527704 +0000 UTC m=+0.126942136 container create e73b634ec9a6950c17ad70261bbd4420de9dc4303df88081cde6e3259cb65a84 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-02-14T17:55:01.593 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:01 smithi017 podman[131174]: 2024-02-14 17:55:01.509784301 +0000 UTC m=+0.313198736 container init e73b634ec9a6950c17ad70261bbd4420de9dc4303df88081cde6e3259cb65a84 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-14T17:55:01.593 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:01 smithi017 podman[131174]: 2024-02-14 17:55:01.513148565 +0000 UTC m=+0.316562993 container start e73b634ec9a6950c17ad70261bbd4420de9dc4303df88081cde6e3259cb65a84 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214) 2024-02-14T17:55:01.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:01 smithi017 podman[131174]: 2024-02-14 17:55:01.592370563 +0000 UTC m=+0.395785003 container attach e73b634ec9a6950c17ad70261bbd4420de9dc4303df88081cde6e3259cb65a84 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-14T17:55:01.940 INFO:journalctl@ceph.osd.0.smithi017.stdout:Feb 14 17:55:01 smithi017 podman[131174]: 2024-02-14 17:55:01.814713094 +0000 UTC m=+0.618127527 container died e73b634ec9a6950c17ad70261bbd4420de9dc4303df88081cde6e3259cb65a84 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-14T17:55:02.343 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.0.service' 2024-02-14T17:55:02.386 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:02.386 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-14T17:55:02.386 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-14T17:55:02.386 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.1 2024-02-14T17:55:02.690 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:02 smithi017 systemd[1]: Stopping Ceph osd.1 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:03.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[101393]: 2024-02-14T17:55:02.740+0000 7f4dfedb1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:55:03.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[101393]: 2024-02-14T17:55:02.740+0000 7f4dfedb1700 -1 osd.1 158 *** Got signal Terminated *** 2024-02-14T17:55:03.190 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:02 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1[101393]: 2024-02-14T17:55:02.740+0000 7f4dfedb1700 -1 osd.1 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:55:04.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:04.269Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:04.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:04 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:04.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:08.334 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:08 smithi017 podman[131281]: 2024-02-14 17:55:08.069069877 +0000 UTC m=+5.450818803 container died 8beec53ad36659ddbbd09cc045502f2460594b781b9c748bb055ecb86c636e33 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:55:08.609 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:08 smithi017 podman[131281]: 2024-02-14 17:55:08.406893112 +0000 UTC m=+5.788642037 container cleanup 8beec53ad36659ddbbd09cc045502f2460594b781b9c748bb055ecb86c636e33 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-14T17:55:08.609 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:08 smithi017 bash[131281]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1 2024-02-14T17:55:08.939 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:08 smithi017 podman[131310]: 2024-02-14 17:55:08.608233845 +0000 UTC m=+0.534756488 container remove 8beec53ad36659ddbbd09cc045502f2460594b781b9c748bb055ecb86c636e33 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-02-14T17:55:09.345 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:09 smithi017 podman[131402]: 2024-02-14 17:55:08.931534609 +0000 UTC m=+0.018197077 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:09.345 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:09 smithi017 podman[131402]: 2024-02-14 17:55:09.059212664 +0000 UTC m=+0.145875169 container create 3992fbcf6dc6f40b46e945ba3b10cf29799c78bf38367d957623ccd27b23b309 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-14T17:55:09.346 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:09 smithi017 podman[131402]: 2024-02-14 17:55:09.2536784 +0000 UTC m=+0.340340861 container init 3992fbcf6dc6f40b46e945ba3b10cf29799c78bf38367d957623ccd27b23b309 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-14T17:55:09.346 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:09 smithi017 podman[131402]: 2024-02-14 17:55:09.25692895 +0000 UTC m=+0.343591419 container start 3992fbcf6dc6f40b46e945ba3b10cf29799c78bf38367d957623ccd27b23b309 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-14T17:55:09.346 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:09 smithi017 podman[131402]: 2024-02-14 17:55:09.345052194 +0000 UTC m=+0.431714679 container attach 3992fbcf6dc6f40b46e945ba3b10cf29799c78bf38367d957623ccd27b23b309 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-14T17:55:09.690 INFO:journalctl@ceph.osd.1.smithi017.stdout:Feb 14 17:55:09 smithi017 podman[131402]: 2024-02-14 17:55:09.563950722 +0000 UTC m=+0.650613182 container died 3992fbcf6dc6f40b46e945ba3b10cf29799c78bf38367d957623ccd27b23b309 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-14T17:55:10.176 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.1.service' 2024-02-14T17:55:10.217 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:10.217 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-14T17:55:10.217 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-14T17:55:10.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.2 2024-02-14T17:55:10.439 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:10 smithi017 systemd[1]: Stopping Ceph osd.2 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:10.939 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[107982]: 2024-02-14T17:55:10.542+0000 7fd436000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:55:10.940 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[107982]: 2024-02-14T17:55:10.542+0000 7fd436000700 -1 osd.2 158 *** Got signal Terminated *** 2024-02-14T17:55:10.940 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:10 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2[107982]: 2024-02-14T17:55:10.542+0000 7fd436000700 -1 osd.2 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:55:13.833 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:13.548Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:13.833 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:13 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:13.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:14.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:14.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:14.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:14.270Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:14.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:14 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:14.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:16.353 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:16 smithi017 podman[131508]: 2024-02-14 17:55:16.061550312 +0000 UTC m=+5.619746257 container died 89cf7ada9e899bd189de1b4cd457bb1619a8d9b38f9689aa1fc2ebef5930327c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-14T17:55:16.689 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:16 smithi017 podman[131508]: 2024-02-14 17:55:16.451147021 +0000 UTC m=+6.009342964 container cleanup 89cf7ada9e899bd189de1b4cd457bb1619a8d9b38f9689aa1fc2ebef5930327c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-14T17:55:16.690 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:16 smithi017 bash[131508]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2 2024-02-14T17:55:17.067 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:16 smithi017 podman[131525]: 2024-02-14 17:55:16.743994246 +0000 UTC m=+0.679546132 container remove 89cf7ada9e899bd189de1b4cd457bb1619a8d9b38f9689aa1fc2ebef5930327c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-14T17:55:17.319 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:17 smithi017 podman[131622]: 2024-02-14 17:55:16.969921046 +0000 UTC m=+0.017870925 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:17.319 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:17 smithi017 podman[131622]: 2024-02-14 17:55:17.08661183 +0000 UTC m=+0.134561743 container create eca270fbc708ddb04ecbd8ad1ddb75a8f256d2d26fc9a59366ced71f0892c652 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-14T17:55:17.319 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:17 smithi017 podman[131622]: 2024-02-14 17:55:17.314329942 +0000 UTC m=+0.362279814 container init eca270fbc708ddb04ecbd8ad1ddb75a8f256d2d26fc9a59366ced71f0892c652 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240214, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-14T17:55:17.319 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:17 smithi017 podman[131622]: 2024-02-14 17:55:17.318262276 +0000 UTC m=+0.366212150 container start eca270fbc708ddb04ecbd8ad1ddb75a8f256d2d26fc9a59366ced71f0892c652 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:55:17.607 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:17 smithi017 podman[131622]: 2024-02-14 17:55:17.413913799 +0000 UTC m=+0.461863681 container attach eca270fbc708ddb04ecbd8ad1ddb75a8f256d2d26fc9a59366ced71f0892c652 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-14T17:55:17.939 INFO:journalctl@ceph.osd.2.smithi017.stdout:Feb 14 17:55:17 smithi017 podman[131622]: 2024-02-14 17:55:17.606588133 +0000 UTC m=+0.654538008 container died eca270fbc708ddb04ecbd8ad1ddb75a8f256d2d26fc9a59366ced71f0892c652 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-02-14T17:55:18.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.2.service' 2024-02-14T17:55:18.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:18.146 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-14T17:55:18.146 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-14T17:55:18.147 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.3 2024-02-14T17:55:18.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:18 smithi017 systemd[1]: Stopping Ceph osd.3 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:18.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:18 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:55:18.469+0000 7f6b11993700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:55:18.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:18 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:55:18.469+0000 7f6b11993700 -1 osd.3 158 *** Got signal Terminated *** 2024-02-14T17:55:18.940 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:18 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:55:18.469+0000 7f6b11993700 -1 osd.3 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:55:21.690 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:21 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:55:21.315+0000 7f6b09f7f700 -1 osd.3 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:58.482301+0000 front 2024-02-14T17:54:58.482331+0000 (oldest deadline 2024-02-14T17:55:20.782557+0000) 2024-02-14T17:55:21.690 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:21.408+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:21.691 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:21.487+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:21.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:21 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:21.690+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:22.689 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:22 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:55:22.307+0000 7f6b09f7f700 -1 osd.3 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:58.482301+0000 front 2024-02-14T17:54:58.482331+0000 (oldest deadline 2024-02-14T17:55:20.782557+0000) 2024-02-14T17:55:22.702 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:22.399+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:22.703 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:22.533+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:22.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:22 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:22.728+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:23.690 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3[114568]: 2024-02-14T17:55:23.308+0000 7f6b09f7f700 -1 osd.3 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:58.482301+0000 front 2024-02-14T17:54:58.482331+0000 (oldest deadline 2024-02-14T17:55:20.782557+0000) 2024-02-14T17:55:23.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:23.548Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:23.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:23.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:23.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:23 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:23.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:23.726 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:23.442+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:23.726 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:23.517+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:23.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:23.725+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:23.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:23 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:23.874+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:24.072 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:23 smithi017 podman[131729]: 2024-02-14 17:55:23.797082381 +0000 UTC m=+5.428209377 container died f3a8d5491fd3ca0f52d515160fa0460ea516d60373b9e3705896538f3f6c6b1c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, ceph=True) 2024-02-14T17:55:24.405 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 podman[131729]: 2024-02-14 17:55:24.169528453 +0000 UTC m=+5.800655448 container cleanup f3a8d5491fd3ca0f52d515160fa0460ea516d60373b9e3705896538f3f6c6b1c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214) 2024-02-14T17:55:24.405 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 bash[131729]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3 2024-02-14T17:55:24.405 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:24.270Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:24.405 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:24.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:24.406 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:24 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:24.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:24.689 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 podman[131746]: 2024-02-14 17:55:24.403934654 +0000 UTC m=+0.609033309 container remove f3a8d5491fd3ca0f52d515160fa0460ea516d60373b9e3705896538f3f6c6b1c (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-14T17:55:24.699 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:24.440+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:24.700 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:24.531+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:24.983 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 podman[131837]: 2024-02-14 17:55:24.621438035 +0000 UTC m=+0.017733871 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:24.983 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 podman[131837]: 2024-02-14 17:55:24.72156755 +0000 UTC m=+0.117863420 container create be7d2748847941675aa423f2251e1372f0185424030b103072f6fa3c85756cd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-14T17:55:24.983 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 podman[131837]: 2024-02-14 17:55:24.899076244 +0000 UTC m=+0.295372078 container init be7d2748847941675aa423f2251e1372f0185424030b103072f6fa3c85756cd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-02-14T17:55:24.983 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 podman[131837]: 2024-02-14 17:55:24.902716923 +0000 UTC m=+0.299012754 container start be7d2748847941675aa423f2251e1372f0185424030b103072f6fa3c85756cd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-02-14T17:55:24.983 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:24 smithi017 podman[131837]: 2024-02-14 17:55:24.982162709 +0000 UTC m=+0.378458538 container attach be7d2748847941675aa423f2251e1372f0185424030b103072f6fa3c85756cd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-02-14T17:55:24.983 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:24.834+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:24.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:24 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:24.699+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:25.440 INFO:journalctl@ceph.osd.3.smithi017.stdout:Feb 14 17:55:25 smithi017 podman[131837]: 2024-02-14 17:55:25.197782155 +0000 UTC m=+0.594077991 container died be7d2748847941675aa423f2251e1372f0185424030b103072f6fa3c85756cd3 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS) 2024-02-14T17:55:25.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.3.service' 2024-02-14T17:55:25.728 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:25.728 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-14T17:55:25.728 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-14T17:55:25.729 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.4 2024-02-14T17:55:25.733 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:25.690+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:25.733 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:25.480+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:25.733 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:25.535+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:26.039 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:25 smithi111 systemd[1]: Stopping Ceph osd.4 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:26.040 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:25 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:25.830+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:26.483 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:26.038+0000 7f524f88c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:55:26.484 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:26.038+0000 7f524f88c700 -1 osd.4 158 *** Got signal Terminated *** 2024-02-14T17:55:26.484 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:26.038+0000 7f524f88c700 -1 osd.4 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:55:26.829 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:26.511+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:26.830 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:26.574+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:26.830 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:26.665+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:27.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:26 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:26.827+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:27.876 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:27.521+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:27.877 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:27.641+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:27.877 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:27.560+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:28.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:27 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:27.874+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:28.867 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:28.500+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:28.868 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:28.518+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:28.868 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:28.642+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:29.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:28 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:28.865+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:29.818 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:29.525+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:29.819 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:29.515+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:29.819 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:29.606+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:30.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:29.817+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:30.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:29 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:29.817+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:30.867 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4[78693]: 2024-02-14T17:55:30.502+0000 7f524be80700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:59.876347+0000 front 2024-02-14T17:54:59.876428+0000 (oldest deadline 2024-02-14T17:55:20.976188+0000) 2024-02-14T17:55:30.867 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:30.549+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:30.867 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:30.549+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:30.868 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:30.582+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:30.868 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:30.582+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:31.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:30.866+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:31.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:30 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:30.866+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:31.716 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:31 smithi111 podman[101790]: 2024-02-14 17:55:31.445504843 +0000 UTC m=+5.514549595 container died c579b5723b701a2e9ad4e6ff6d29c36618e7715d2661821834e20d0d171a2b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:55:31.716 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:31.587+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:31.716 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:31.587+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:31.717 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:31.580+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:31.717 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:31.580+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:31.983 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:31 smithi111 podman[101790]: 2024-02-14 17:55:31.872906776 +0000 UTC m=+5.941951582 container cleanup c579b5723b701a2e9ad4e6ff6d29c36618e7715d2661821834e20d0d171a2b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214) 2024-02-14T17:55:31.984 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:31 smithi111 bash[101790]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4 2024-02-14T17:55:31.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:31.879+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:31.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:31 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:31.879+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:32.483 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:32 smithi111 podman[101807]: 2024-02-14 17:55:32.074303261 +0000 UTC m=+0.628236267 container remove c579b5723b701a2e9ad4e6ff6d29c36618e7715d2661821834e20d0d171a2b45 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-14T17:55:32.778 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:32.553+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:32.778 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:32.553+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:32.778 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:32.629+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:32.778 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:32.629+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:32.779 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:32 smithi111 podman[101898]: 2024-02-14 17:55:32.393037575 +0000 UTC m=+0.018151212 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:32.779 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:32 smithi111 podman[101898]: 2024-02-14 17:55:32.491623476 +0000 UTC m=+0.116737118 container create dad4758fbbb132fadd853b8c63ea65b2416eabefbb9600651bd9ded03ef1c739 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-14T17:55:32.779 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:32 smithi111 podman[101898]: 2024-02-14 17:55:32.702521862 +0000 UTC m=+0.327635500 container init dad4758fbbb132fadd853b8c63ea65b2416eabefbb9600651bd9ded03ef1c739 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS) 2024-02-14T17:55:32.779 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:32 smithi111 podman[101898]: 2024-02-14 17:55:32.705601572 +0000 UTC m=+0.330715213 container start dad4758fbbb132fadd853b8c63ea65b2416eabefbb9600651bd9ded03ef1c739 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-14T17:55:32.779 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:32 smithi111 podman[101898]: 2024-02-14 17:55:32.777727237 +0000 UTC m=+0.402840891 container attach dad4758fbbb132fadd853b8c63ea65b2416eabefbb9600651bd9ded03ef1c739 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-14T17:55:33.234 INFO:journalctl@ceph.osd.4.smithi111.stdout:Feb 14 17:55:32 smithi111 podman[101898]: 2024-02-14 17:55:32.992397078 +0000 UTC m=+0.617510721 container died dad4758fbbb132fadd853b8c63ea65b2416eabefbb9600651bd9ded03ef1c739 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:55:33.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:32.915+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:33.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:32 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:32.915+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:33.619 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:33.528+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:33.620 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:33.528+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:33.737 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.4.service' 2024-02-14T17:55:33.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:33.817 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-14T17:55:33.818 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-14T17:55:33.818 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.5 2024-02-14T17:55:33.881 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:33.617+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:33.881 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:33.617+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:33.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:33.549Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:33.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:33 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:33.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:34.138 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:33 smithi111 systemd[1]: Stopping Ceph osd.5 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:34.138 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:33.962+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:34.138 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:33 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:33.962+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:34.483 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:34.226+0000 7f944b777700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:55:34.484 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:34.226+0000 7f944b777700 -1 osd.5 158 *** Got signal Terminated *** 2024-02-14T17:55:34.484 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:34.226+0000 7f944b777700 -1 osd.5 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:55:34.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:34.271Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:34.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:34.271Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:34.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:34 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:34.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:34.920 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:34.497+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:34.920 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:34.497+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:34.921 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:34.660+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:34.921 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:34.660+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:35.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:34.917+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:35.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:34 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:34.917+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:35.880 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:35.671+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:35.880 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:35.671+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:35.880 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:35.495+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:35.880 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:35.495+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:36.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:35.878+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:36.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:35 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:35.878+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:36.875 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:36 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:36.502+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:36.875 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:36 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:36.502+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:36.876 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:36 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:36.714+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:36.876 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:36 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:36.714+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:37.233 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:36 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:36.874+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:37.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:36 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:36.874+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:37.733 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:37.471+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:37.733 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:37.471+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:38.233 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:37.761+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:38.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:37.761+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:38.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:37.761+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:38.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:37.876+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:38.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:37.876+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:38.234 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:37 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:37.876+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:14.631386+0000 front 2024-02-14T17:55:14.631339+0000 (oldest deadline 2024-02-14T17:55:37.531217+0000) 2024-02-14T17:55:38.728 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:38.444+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:38.728 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:38.444+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:38.983 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:38.856+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:57.129221+0000 front 2024-02-14T17:54:57.129210+0000 (oldest deadline 2024-02-14T17:55:23.028992+0000) 2024-02-14T17:55:38.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:38.856+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:07.130098+0000 front 2024-02-14T17:55:07.130015+0000 (oldest deadline 2024-02-14T17:55:29.429859+0000) 2024-02-14T17:55:38.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5[83657]: 2024-02-14T17:55:38.856+0000 7f9443d63700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:14.631386+0000 front 2024-02-14T17:55:14.631339+0000 (oldest deadline 2024-02-14T17:55:37.531217+0000) 2024-02-14T17:55:38.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:38.727+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:38.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:38.727+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:38.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:38 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:38.727+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:39.704 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:39 smithi111 podman[102007]: 2024-02-14 17:55:39.662199014 +0000 UTC m=+5.588449624 container died 99350ba8e2aa6b7369ae09c94421d252999ac4357833ff2b76563b2e10230623 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-14T17:55:39.705 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:39 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:39.414+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:39.705 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:39 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:39.414+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:39.705 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:39 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:39.414+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:39.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:39 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:39.703+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:39.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:39 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:39.703+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:39.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:39 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:39.703+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:40.398 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:39 smithi111 podman[102007]: 2024-02-14 17:55:39.985693947 +0000 UTC m=+5.911944608 container cleanup 99350ba8e2aa6b7369ae09c94421d252999ac4357833ff2b76563b2e10230623 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9) 2024-02-14T17:55:40.399 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:39 smithi111 bash[102007]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5 2024-02-14T17:55:40.399 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:40 smithi111 podman[102025]: 2024-02-14 17:55:40.194968229 +0000 UTC m=+0.530044086 container remove 99350ba8e2aa6b7369ae09c94421d252999ac4357833ff2b76563b2e10230623 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-02-14T17:55:40.694 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:40 smithi111 podman[102117]: 2024-02-14 17:55:40.506711829 +0000 UTC m=+0.018213762 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:40.694 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:40 smithi111 podman[102117]: 2024-02-14 17:55:40.612351291 +0000 UTC m=+0.123853219 container create 2b5609e9e80162848987f0eb7547e87302ba044c5d139bab8d426547b993c0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-02-14T17:55:40.695 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:40.396+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:40.695 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:40.396+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:40.695 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:40.396+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:40.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:40 smithi111 podman[102117]: 2024-02-14 17:55:40.814804529 +0000 UTC m=+0.326306468 container init 2b5609e9e80162848987f0eb7547e87302ba044c5d139bab8d426547b993c0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-14T17:55:40.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:40 smithi111 podman[102117]: 2024-02-14 17:55:40.817940981 +0000 UTC m=+0.329442911 container start 2b5609e9e80162848987f0eb7547e87302ba044c5d139bab8d426547b993c0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240214) 2024-02-14T17:55:40.984 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:40 smithi111 podman[102117]: 2024-02-14 17:55:40.889797745 +0000 UTC m=+0.401299687 container attach 2b5609e9e80162848987f0eb7547e87302ba044c5d139bab8d426547b993c0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-14T17:55:40.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:40.693+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:40.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:40.693+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:40.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:40 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:40.693+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:41.370 INFO:journalctl@ceph.osd.5.smithi111.stdout:Feb 14 17:55:41 smithi111 podman[102117]: 2024-02-14 17:55:41.106937338 +0000 UTC m=+0.618439272 container died 2b5609e9e80162848987f0eb7547e87302ba044c5d139bab8d426547b993c0b0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240214, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-14T17:55:41.659 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:41.369+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:41.660 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:41.369+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:41.660 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:41.369+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:41.687 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.5.service' 2024-02-14T17:55:41.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:41.770 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-14T17:55:41.770 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-14T17:55:41.770 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.6 2024-02-14T17:55:41.983 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:41 smithi111 systemd[1]: Stopping Ceph osd.6 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:41.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:41.697+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:41.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:41.697+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:41.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:41 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:41.697+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:42.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:42.095+0000 7f030d995700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:55:42.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:42.095+0000 7f030d995700 -1 osd.6 158 *** Got signal Terminated *** 2024-02-14T17:55:42.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:42.095+0000 7f030d995700 -1 osd.6 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:55:42.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:42.322+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:42.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:42.322+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:42.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:42.322+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:42.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:42.673+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:42.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:42.673+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:42.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:42 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:42.673+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:43.588 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:43 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:43.324+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:43.588 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:43 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:43.324+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:43.589 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:43 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:43.324+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:43.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:43.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:43.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:43 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:43.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:43.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:43 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:43.700+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:43.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:43 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:43.700+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:43.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:43 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:43.700+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:44.483 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:44.317+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:44.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:44.317+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:44.484 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:44.317+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:44.608 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:44.272Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:44.609 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:44.272Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:44.609 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:44 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:44.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:44.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:44.697+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:44.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:44.697+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:44.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:44 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:44.697+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:45.711 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:45 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:45.355+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:45.711 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:45 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:45.355+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:45.711 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:45 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:45.355+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:45.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:45 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:45.710+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:45.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:45 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:45.710+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:45.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:45 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:45.710+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:46.670 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:46 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:46.340+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:55.925020+0000 front 2024-02-14T17:54:55.924983+0000 (oldest deadline 2024-02-14T17:55:20.624853+0000) 2024-02-14T17:55:46.671 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:46 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:46.340+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:05.826237+0000 front 2024-02-14T17:55:05.826146+0000 (oldest deadline 2024-02-14T17:55:30.526066+0000) 2024-02-14T17:55:46.671 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:46 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6[88205]: 2024-02-14T17:55:46.340+0000 7f0309f89700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.927434+0000 front 2024-02-14T17:55:13.927347+0000 (oldest deadline 2024-02-14T17:55:39.227253+0000) 2024-02-14T17:55:46.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:46 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:46.670+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:46.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:46 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:46.670+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:46.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:46 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:46.670+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:47.700 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:47 smithi111 podman[102223]: 2024-02-14 17:55:47.445785632 +0000 UTC m=+5.457409594 container died f5aa07b95e17a9eaca0d17b0a1617b9f2c03fcd820c32a46646f7ad5c3fcd6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-14T17:55:47.972 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:47 smithi111 podman[102223]: 2024-02-14 17:55:47.787383307 +0000 UTC m=+5.799007268 container cleanup f5aa07b95e17a9eaca0d17b0a1617b9f2c03fcd820c32a46646f7ad5c3fcd6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-14T17:55:47.972 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:47 smithi111 bash[102223]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6 2024-02-14T17:55:47.972 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:47.698+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:47.972 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:47.698+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:47.972 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:47 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:47.698+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:48.233 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:47 smithi111 podman[102241]: 2024-02-14 17:55:47.971849896 +0000 UTC m=+0.515969533 container remove f5aa07b95e17a9eaca0d17b0a1617b9f2c03fcd820c32a46646f7ad5c3fcd6d4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-14T17:55:48.667 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:48 smithi111 podman[102333]: 2024-02-14 17:55:48.389464015 +0000 UTC m=+0.097851798 container create cc859ed1e353596d8dabc34169bacd7f862a53ca8d665b83126d75a85222f307 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-14T17:55:48.667 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:48 smithi111 podman[102333]: 2024-02-14 17:55:48.310165288 +0000 UTC m=+0.018553079 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:48.667 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:48 smithi111 podman[102333]: 2024-02-14 17:55:48.583294644 +0000 UTC m=+0.291682427 container init cc859ed1e353596d8dabc34169bacd7f862a53ca8d665b83126d75a85222f307 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, ceph=True, org.label-schema.license=GPLv2) 2024-02-14T17:55:48.667 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:48 smithi111 podman[102333]: 2024-02-14 17:55:48.58625044 +0000 UTC m=+0.294638230 container start cc859ed1e353596d8dabc34169bacd7f862a53ca8d665b83126d75a85222f307 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-14T17:55:48.667 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:48 smithi111 podman[102333]: 2024-02-14 17:55:48.666614395 +0000 UTC m=+0.375002198 container attach cc859ed1e353596d8dabc34169bacd7f862a53ca8d665b83126d75a85222f307 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240214, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-02-14T17:55:48.983 INFO:journalctl@ceph.osd.6.smithi111.stdout:Feb 14 17:55:48 smithi111 podman[102333]: 2024-02-14 17:55:48.868855479 +0000 UTC m=+0.577243263 container died cc859ed1e353596d8dabc34169bacd7f862a53ca8d665b83126d75a85222f307 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:55:48.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:48.686+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:48.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:48.686+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:48.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:48.686+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:48.984 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:48 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:48.686+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-02-14T17:55:22.676698+0000 front 2024-02-14T17:55:22.676723+0000 (oldest deadline 2024-02-14T17:55:47.976647+0000) 2024-02-14T17:55:49.573 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.6.service' 2024-02-14T17:55:49.617 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:49.617 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-14T17:55:49.617 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-14T17:55:49.618 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.7 2024-02-14T17:55:49.940 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 systemd[1]: Stopping Ceph osd.7 for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:49.940 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:49.725+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:49.940 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:49.725+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:49.941 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:49.725+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:49.941 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:49.725+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-02-14T17:55:22.676698+0000 front 2024-02-14T17:55:22.676723+0000 (oldest deadline 2024-02-14T17:55:47.976647+0000) 2024-02-14T17:55:49.941 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:49.939+0000 7fed57220700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-14T17:55:49.941 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:49.939+0000 7fed57220700 -1 osd.7 158 *** Got signal Terminated *** 2024-02-14T17:55:49.941 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:49 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:49.939+0000 7fed57220700 -1 osd.7 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-14T17:55:51.135 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:50.766+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:51.135 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:50.767+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:51.135 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:50.767+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:51.135 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:50 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:50.767+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-02-14T17:55:22.676698+0000 front 2024-02-14T17:55:22.676723+0000 (oldest deadline 2024-02-14T17:55:47.976647+0000) 2024-02-14T17:55:52.151 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:51.804+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:52.151 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:51.804+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:52.151 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:51.804+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:52.151 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:51 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:51.804+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-02-14T17:55:22.676698+0000 front 2024-02-14T17:55:22.676723+0000 (oldest deadline 2024-02-14T17:55:47.976647+0000) 2024-02-14T17:55:53.233 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:52.837+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:53.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:52.837+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:53.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:52.837+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:53.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:52.837+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-02-14T17:55:22.676698+0000 front 2024-02-14T17:55:22.676723+0000 (oldest deadline 2024-02-14T17:55:47.976647+0000) 2024-02-14T17:55:53.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:52 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:52.837+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.111:6804 osd.4 since back 2024-02-14T17:55:30.777979+0000 front 2024-02-14T17:55:30.777997+0000 (oldest deadline 2024-02-14T17:55:52.478004+0000) 2024-02-14T17:55:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:53.549Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:53.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:53.940 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:53 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:53.550Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:54.233 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:53.867+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:54.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:53.867+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:54.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:53.867+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:54.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:53.867+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-02-14T17:55:22.676698+0000 front 2024-02-14T17:55:22.676723+0000 (oldest deadline 2024-02-14T17:55:47.976647+0000) 2024-02-14T17:55:54.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:53 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:53.867+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.111:6804 osd.4 since back 2024-02-14T17:55:30.777979+0000 front 2024-02-14T17:55:30.777997+0000 (oldest deadline 2024-02-14T17:55:52.478004+0000) 2024-02-14T17:55:54.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:54.273Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:54.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:54.273Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi017.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.17:8443: connect: connection refused" 2024-02-14T17:55:54.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:55:54 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:55:54.274Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi111.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.111:8443: connect: connection refused" 2024-02-14T17:55:55.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:54.903+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6806 osd.0 since back 2024-02-14T17:54:54.072778+0000 front 2024-02-14T17:54:54.072885+0000 (oldest deadline 2024-02-14T17:55:21.673370+0000) 2024-02-14T17:55:55.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:54.903+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6814 osd.1 since back 2024-02-14T17:55:06.974435+0000 front 2024-02-14T17:55:06.974357+0000 (oldest deadline 2024-02-14T17:55:29.874396+0000) 2024-02-14T17:55:55.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:54.903+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6822 osd.2 since back 2024-02-14T17:55:13.975266+0000 front 2024-02-14T17:55:13.975284+0000 (oldest deadline 2024-02-14T17:55:37.475269+0000) 2024-02-14T17:55:55.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:54.903+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.17:6830 osd.3 since back 2024-02-14T17:55:22.676698+0000 front 2024-02-14T17:55:22.676723+0000 (oldest deadline 2024-02-14T17:55:47.976647+0000) 2024-02-14T17:55:55.234 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:54 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7[92894]: 2024-02-14T17:55:54.903+0000 7fed4f80c700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.111:6804 osd.4 since back 2024-02-14T17:55:30.777979+0000 front 2024-02-14T17:55:30.777997+0000 (oldest deadline 2024-02-14T17:55:52.478004+0000) 2024-02-14T17:55:55.724 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:55 smithi111 podman[102441]: 2024-02-14 17:55:55.467319705 +0000 UTC m=+5.625243527 container died 3685781bd0e6b63465428be9b01c8abb4805f51772f8d2579de83c4ed7ac79d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-14T17:55:55.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:55 smithi111 podman[102441]: 2024-02-14 17:55:55.906476792 +0000 UTC m=+6.064400658 container cleanup 3685781bd0e6b63465428be9b01c8abb4805f51772f8d2579de83c4ed7ac79d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS) 2024-02-14T17:55:55.983 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:55 smithi111 bash[102441]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7 2024-02-14T17:55:56.483 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:56 smithi111 podman[102470]: 2024-02-14 17:55:56.099624049 +0000 UTC m=+0.631919205 container remove 3685781bd0e6b63465428be9b01c8abb4805f51772f8d2579de83c4ed7ac79d6 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-14T17:55:56.774 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:56 smithi111 podman[102565]: 2024-02-14 17:55:56.492052845 +0000 UTC m=+0.110021642 container create b7efb031edd33580b9eca0c53d28acd807afc288078e41bd1f091b7ac6f2d789 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-14T17:55:56.774 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:56 smithi111 podman[102565]: 2024-02-14 17:55:56.399474611 +0000 UTC m=+0.017443419 image pull 170846cf6432fa177c159b5d8a38454ab2459d562a59c58b9aa838f6e31ddf86 quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b 2024-02-14T17:55:56.774 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:56 smithi111 podman[102565]: 2024-02-14 17:55:56.73609386 +0000 UTC m=+0.354062667 container init b7efb031edd33580b9eca0c53d28acd807afc288078e41bd1f091b7ac6f2d789 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-14T17:55:56.774 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:56 smithi111 podman[102565]: 2024-02-14 17:55:56.739328787 +0000 UTC m=+0.357297585 container start b7efb031edd33580b9eca0c53d28acd807afc288078e41bd1f091b7ac6f2d789 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-14T17:55:57.029 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:56 smithi111 podman[102565]: 2024-02-14 17:55:56.798515688 +0000 UTC m=+0.416484498 container attach b7efb031edd33580b9eca0c53d28acd807afc288078e41bd1f091b7ac6f2d789 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-14T17:55:57.029 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:56.774Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:55:57.029 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:56.776Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:55:57.029 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:56.777Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:55:57.029 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:56 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:56.778Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.17:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.17:8765: connect: connection refused" 2024-02-14T17:55:57.297 INFO:journalctl@ceph.osd.7.smithi111.stdout:Feb 14 17:55:57 smithi111 podman[102565]: 2024-02-14 17:55:57.028806865 +0000 UTC m=+0.646775667 container died b7efb031edd33580b9eca0c53d28acd807afc288078e41bd1f091b7ac6f2d789 (image=quay.ceph.io/ceph-ci/ceph@sha256:e74cf8e1e43e8dfa713eb599fa0104396b2e29af34a3f40d868498c8111bdb4b, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-osd-7-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-02-14-1033-8727ce9, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-14T17:55:57.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@osd.7.service' 2024-02-14T17:55:57.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:57.692 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-14T17:55:57.692 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-14T17:55:57.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a 2024-02-14T17:55:57.983 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:57 smithi111 systemd[1]: Stopping Ceph prometheus.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-14T17:55:58.311 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a[64376]: ts=2024-02-14T17:55:58.026Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-14T17:55:58.312 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 podman[102671]: 2024-02-14 17:55:58.036654663 +0000 UTC m=+0.127654687 container died 51b0c15c913d518f1ca2e9ebfb8a242ad10871088f356a53754c055665231c75 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:55:58.652 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 podman[102671]: 2024-02-14 17:55:58.425996008 +0000 UTC m=+0.516996032 container cleanup 51b0c15c913d518f1ca2e9ebfb8a242ad10871088f356a53754c055665231c75 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-14T17:55:58.652 INFO:journalctl@ceph.prometheus.a.smithi111.stdout:Feb 14 17:55:58 smithi111 bash[102671]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-prometheus-a 2024-02-14T17:55:58.792 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-612435b4-cb5c-11ee-95b9-87774f69a715@prometheus.a.service' 2024-02-14T17:55:58.828 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-14T17:55:58.829 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-14T17:55:58.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-14T17:56:00.690 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:00 smithi017 systemd[1]: Stopping Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:56:00.690 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:00 smithi017 podman[132227]: 2024-02-14 17:56:00.571886122 +0000 UTC m=+0.106659055 container died 3ed6d4a4c5ad2dbc911bc3d2fbb9a2fa17f2766a451cda0324c97b5d6a77e68f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:56:01.095 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:00 smithi017 podman[132227]: 2024-02-14 17:56:00.835930624 +0000 UTC m=+0.370703554 container cleanup 3ed6d4a4c5ad2dbc911bc3d2fbb9a2fa17f2766a451cda0324c97b5d6a77e68f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:56:01.096 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:00 smithi017 bash[132227]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a 2024-02-14T17:56:01.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:01 smithi017 podman[132241]: 2024-02-14 17:56:01.095088924 +0000 UTC m=+0.527322042 container remove 3ed6d4a4c5ad2dbc911bc3d2fbb9a2fa17f2766a451cda0324c97b5d6a77e68f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-14T17:56:01.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:01 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:56:01.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:01 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-14T17:56:01.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:01 smithi017 systemd[1]: Stopped Ceph node-exporter.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:56:01.440 INFO:journalctl@ceph.node-exporter.a.smithi017.stdout:Feb 14 17:56:01 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.a.service: Consumed 3.195s CPU time. 2024-02-14T17:56:01.741 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:01 smithi017 systemd[1]: Stopping Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:56:01.741 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:01 smithi017 ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a[78655]: ts=2024-02-14T17:56:01.689Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-14T17:56:02.025 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:01 smithi017 podman[132345]: 2024-02-14 17:56:01.741027649 +0000 UTC m=+0.155707942 container died 3ba623745a7f49969c82265b7e36e9b5edf28612a2e4c0611763db503416f0c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:56:02.358 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:02 smithi017 podman[132345]: 2024-02-14 17:56:02.106725434 +0000 UTC m=+0.521405726 container cleanup 3ba623745a7f49969c82265b7e36e9b5edf28612a2e4c0611763db503416f0c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:56:02.359 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:02 smithi017 bash[132345]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a 2024-02-14T17:56:02.359 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:02 smithi017 podman[132359]: 2024-02-14 17:56:02.274252131 +0000 UTC m=+0.538319437 container remove 3ba623745a7f49969c82265b7e36e9b5edf28612a2e4c0611763db503416f0c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-14T17:56:02.359 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:02 smithi017 podman[132359]: 2024-02-14 17:56:02.358211908 +0000 UTC m=+0.622279214 volume remove 8b8e97e1e6197a0da2fe63f7d0b6ff67d5ef27a3ff3b36c852305ea7adf36ac4 2024-02-14T17:56:02.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:02 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-14T17:56:02.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:02 smithi017 systemd[1]: Stopped Ceph alertmanager.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:56:02.690 INFO:journalctl@ceph.alertmanager.a.smithi017.stdout:Feb 14 17:56:02 smithi017 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@alertmanager.a.service: Consumed 1.668s CPU time. 2024-02-14T17:56:25.546 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 --force --keep-logs 2024-02-14T17:56:27.234 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:26 smithi111 systemd[1]: Stopping Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:56:27.234 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 podman[103068]: 2024-02-14 17:56:27.171450379 +0000 UTC m=+0.107063966 container died b2920c9965f16f1af7c8db068ede6929f67644f555ba4d5a5cf995a9f081bd72 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:56:27.733 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 podman[103068]: 2024-02-14 17:56:27.535918739 +0000 UTC m=+0.471532309 container cleanup b2920c9965f16f1af7c8db068ede6929f67644f555ba4d5a5cf995a9f081bd72 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:56:27.734 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 bash[103068]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b 2024-02-14T17:56:28.132 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 podman[103082]: 2024-02-14 17:56:27.829222052 +0000 UTC m=+0.658597602 container remove b2920c9965f16f1af7c8db068ede6929f67644f555ba4d5a5cf995a9f081bd72 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-14T17:56:28.133 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:56:28.133 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-14T17:56:28.133 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 systemd[1]: Stopped Ceph node-exporter.b for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:56:28.133 INFO:journalctl@ceph.node-exporter.b.smithi111.stdout:Feb 14 17:56:27 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@node-exporter.b.service: Consumed 3.099s CPU time. 2024-02-14T17:56:28.401 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:28 smithi111 systemd[1]: Stopping Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715... 2024-02-14T17:56:28.698 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:28 smithi111 podman[103202]: 2024-02-14 17:56:28.401065192 +0000 UTC m=+0.107917612 container died 7b859cc99d088a9e3fd12227c65865b821610bd61018f692067dde307d3d4e02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, build-date=2022-04-29T11:48:11.055897, name=ubi8) 2024-02-14T17:56:28.983 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:28 smithi111 podman[103202]: 2024-02-14 17:56:28.698949642 +0000 UTC m=+0.405802074 container cleanup 7b859cc99d088a9e3fd12227c65865b821610bd61018f692067dde307d3d4e02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, vcs-type=git, io.openshift.expose-services=, distribution-scope=public, maintainer=Paul Cuzner , architecture=x86_64, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0) 2024-02-14T17:56:28.984 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:28 smithi111 bash[103202]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a 2024-02-14T17:56:28.984 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:28 smithi111 podman[103216]: 2024-02-14 17:56:28.874836801 +0000 UTC m=+0.476459929 container remove 7b859cc99d088a9e3fd12227c65865b821610bd61018f692067dde307d3d4e02 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-612435b4-cb5c-11ee-95b9-87774f69a715-grafana-a, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.openshift.tags=base rhel8, name=ubi8, maintainer=Paul Cuzner , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container) 2024-02-14T17:56:28.984 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:28 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-14T17:56:29.483 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:29 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-14T17:56:29.484 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:29 smithi111 systemd[1]: Stopped Ceph grafana.a for 612435b4-cb5c-11ee-95b9-87774f69a715. 2024-02-14T17:56:29.484 INFO:journalctl@ceph.grafana.a.smithi111.stdout:Feb 14 17:56:29 smithi111 systemd[1]: ceph-612435b4-cb5c-11ee-95b9-87774f69a715@grafana.a.service: Consumed 9.307s CPU time. 2024-02-14T17:56:41.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:56:41.192 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-14T17:56:41.222 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-14T17:56:41.225 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762/remote/smithi017/crash 2024-02-14T17:56:41.226 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/crash -- . 2024-02-14T17:56:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:tar: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-14T17:56:41.277 INFO:teuthology.orchestra.run.smithi017.stderr:tar: Error is not recoverable: exiting now 2024-02-14T17:56:41.278 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762/remote/smithi111/crash 2024-02-14T17:56:41.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/crash -- . 2024-02-14T17:56:41.319 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-14T17:56:41.319 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-02-14T17:56:41.320 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-14T17:56:41.321 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:56:41.351 INFO:teuthology.orchestra.run.smithi017.stdout:2024-02-14T17:15:50.908497+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:56:41.352 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-14T17:56:41.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[SEC\]' /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:56:41.422 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]' /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:56:41.491 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[WRN\]' /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-14T17:56:41.559 INFO:teuthology.orchestra.run.smithi017.stdout:2024-02-14T17:15:50.908497+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-14T17:56:41.560 INFO:tasks.cephadm:Compressing logs... 2024-02-14T17:56:41.560 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-14T17:56:41.602 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-14T17:56:41.629 INFO:teuthology.orchestra.run.smithi017.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-14T17:56:41.629 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-14T17:56:41.629 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.a.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi017.vbwecr.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.a.log: /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi017.vbwecr.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.1.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr: 76.2% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi017.vbwecr.log.gz 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.3.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.audit.log: /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mgr.y.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/tcmu-runner.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.0.log 2024-02-14T17:56:41.630 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/tcmu-runner.log: 87.9% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/tcmu-runner.log.gz 2024-02-14T17:56:41.631 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:: No such file or directory 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.6.log 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi111.zvlcwh.log/var/log/ceph/cephadm.log: 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.audit.log 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi111.zvlcwh.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.4.log 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr: 76.3% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-client.rgw.foo.smithi111.zvlcwh.log.gz 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.7.log 2024-02-14T17:56:41.635 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.audit.log: /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.5.log 2024-02-14T17:56:41.636 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mgr.x.log 2024-02-14T17:56:41.636 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.cephadm.log 2024-02-14T17:56:41.636 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log 2024-02-14T17:56:41.636 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.cephadm.log: 90.9% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-14T17:56:41.636 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-14T17:56:41.637 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log: 91.0% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-14T17:56:41.638 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log 2024-02-14T17:56:41.638 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.b.log 2024-02-14T17:56:41.639 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.cephadm.log: 94.4% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.audit.log.gz 2024-02-14T17:56:41.639 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-volume.log: 89.1% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-14T17:56:41.640 INFO:teuthology.orchestra.run.smithi017.stderr: 94.5% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.cephadm.log.gz 2024-02-14T17:56:41.643 INFO:teuthology.orchestra.run.smithi017.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-14T17:56:41.644 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.b.log: 86.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-14T17:56:41.650 INFO:teuthology.orchestra.run.smithi111.stderr: 91.1% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-14T17:56:41.652 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.2.log 2024-02-14T17:56:41.652 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-14T17:56:41.659 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.2.log: 93.8% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph.log.gz 2024-02-14T17:56:41.660 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.c.log 2024-02-14T17:56:41.676 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-volume.log: /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-14T17:56:41.779 INFO:teuthology.orchestra.run.smithi111.stderr: 89.5% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mgr.x.log.gz 2024-02-14T17:56:42.130 INFO:teuthology.orchestra.run.smithi017.stderr: 89.0% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mgr.y.log.gz 2024-02-14T17:56:42.263 INFO:teuthology.orchestra.run.smithi111.stderr: 92.4% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.b.log.gz 2024-02-14T17:56:42.336 INFO:teuthology.orchestra.run.smithi017.stderr: 92.5% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.c.log.gz 2024-02-14T17:56:43.144 INFO:teuthology.orchestra.run.smithi111.stderr: 93.1% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.6.log.gz 2024-02-14T17:56:43.311 INFO:teuthology.orchestra.run.smithi017.stderr: 93.3% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.2.log.gz 2024-02-14T17:56:43.538 INFO:teuthology.orchestra.run.smithi111.stderr: 93.7% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.7.log.gz 2024-02-14T17:56:43.661 INFO:teuthology.orchestra.run.smithi017.stderr: 91.0% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-mon.a.log.gz 2024-02-14T17:56:43.748 INFO:teuthology.orchestra.run.smithi017.stderr: 93.4% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.1.log.gz 2024-02-14T17:56:43.813 INFO:teuthology.orchestra.run.smithi111.stderr: 93.3% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.5.log.gz 2024-02-14T17:56:43.887 INFO:teuthology.orchestra.run.smithi111.stderr: 93.5% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.4.log.gz 2024-02-14T17:56:43.889 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-14T17:56:43.889 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m2.273s 2024-02-14T17:56:43.889 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m8.339s 2024-02-14T17:56:43.889 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.351s 2024-02-14T17:56:43.957 INFO:teuthology.orchestra.run.smithi017.stderr: 93.5% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.0.log.gz 2024-02-14T17:56:44.107 INFO:teuthology.orchestra.run.smithi017.stderr: 93.4% -- replaced with /var/log/ceph/612435b4-cb5c-11ee-95b9-87774f69a715/ceph-osd.3.log.gz 2024-02-14T17:56:44.109 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-02-14T17:56:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m2.495s 2024-02-14T17:56:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:user 0m11.412s 2024-02-14T17:56:44.109 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m0.397s 2024-02-14T17:56:44.109 INFO:tasks.cephadm:Archiving logs... 2024-02-14T17:56:44.109 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762/remote/smithi017/log 2024-02-14T17:56:44.110 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-14T17:56:45.652 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762/remote/smithi111/log 2024-02-14T17:56:45.653 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-14T17:56:46.960 INFO:tasks.cephadm:Removing cluster... 2024-02-14T17:56:46.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 --force 2024-02-14T17:56:47.261 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 612435b4-cb5c-11ee-95b9-87774f69a715 --force 2024-02-14T17:56:47.555 INFO:tasks.cephadm:Removing cephadm ... 2024-02-14T17:56:47.555 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-14T17:56:47.572 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-14T17:56:47.587 INFO:tasks.cephadm:Teardown complete 2024-02-14T17:56:47.587 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-14T17:56:47.597 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-14T17:56:47.597 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:56:47.614 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-14T17:56:47.626 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:56:47.630 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:56:47.630 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-02-14T17:56:47.630 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +2511us[+2489us] +/- 106ms 2024-02-14T17:56:47.630 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -635us[ -657us] +/- 55ms 2024-02-14T17:56:47.630 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 -794us[ -794us] +/- 75ms 2024-02-14T17:56:47.631 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:56:47.641 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-02-14T17:56:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-14T17:56:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-02-14T17:56:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:^- hv01.front.sepia.ceph.com 2 6 377 2 +3177us[+3178us] +/- 107ms 2024-02-14T17:56:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 +61us[ +62us] +/- 55ms 2024-02-14T17:56:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -86us[ -85us] +/- 75ms 2024-02-14T17:56:47.645 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-14T17:56:47.646 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-14T17:56:47.658 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-14T17:56:47.659 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-14T17:56:47.681 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-14T17:56:47.776 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-14T17:56:47.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:56:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:56:47.887 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-14T17:56:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:56:47.918 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-14T17:56:47.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-14T17:56:47.963 INFO:teuthology.task.internal:Duration was 2853.455050 seconds 2024-02-14T17:56:47.964 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-14T17:56:47.997 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-14T17:56:47.997 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-14T17:56:48.000 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-14T17:56:48.042 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:56:48.043 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-14T17:56:48.477 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-14T17:56:48.477 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-02-14T17:56:48.478 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-14T17:56:48.537 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-02-14T17:56:48.537 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-14T17:56:48.565 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-14T17:56:48.565 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-14T17:56:48.580 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-14T17:56:48.639 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-14T17:56:48.639 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-14T17:56:48.649 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-14T17:56:49.237 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-14T17:56:49.252 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-14T17:56:49.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-14T17:56:49.279 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-14T17:56:49.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-14T17:56:49.322 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:56:49.329 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:56:49.351 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-02-14T17:56:49.374 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-02-14T17:56:49.396 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:56:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:56:49.433 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-14T17:56:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-14T17:56:49.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-14T17:56:49.460 INFO:teuthology.task.internal:Transferring archived files... 2024-02-14T17:56:49.461 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762/remote/smithi017 2024-02-14T17:56:49.461 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-14T17:56:49.533 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762/remote/smithi111 2024-02-14T17:56:49.533 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-14T17:56:49.579 INFO:teuthology.task.internal:Removing archive directory... 2024-02-14T17:56:49.580 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-14T17:56:49.582 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-14T17:56:49.626 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-14T17:56:49.639 INFO:teuthology.task.internal:Not uploading archives. 2024-02-14T17:56:49.639 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-14T17:56:49.670 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-14T17:56:49.670 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-14T17:56:49.674 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-14T17:56:49.686 INFO:teuthology.orchestra.run.smithi017.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 14 17:56 /home/ubuntu/cephtest 2024-02-14T17:56:49.693 INFO:teuthology.orchestra.run.smithi111.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 14 17:56 /home/ubuntu/cephtest 2024-02-14T17:56:49.695 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-14T17:56:49.705 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-14T17:56:49.765 INFO:teuthology.nuke:Checking targets against current locks 2024-02-14T17:56:49.795 DEBUG:teuthology.nuke:shortname: smithi017 2024-02-14T17:56:49.796 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-14T17:56:49.820 DEBUG:teuthology.nuke:shortname: smithi111 2024-02-14T17:56:49.820 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-14T17:56:49.825 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 16:59:55.993724', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLlgQNIOt312vnrlk/zAL3nwp0I8wQzfpfijrVc76T3z87q3Ip+MOw2OXO5VHKIV67nkUS2ecXCUHf+mUO77nHM='} 2024-02-14T17:56:49.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-14_13:54:10-orch:cephadm-wip-guits-testing-2024-02-14-1033-distro-default-smithi/7559762', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-14 16:59:55.991912', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJsK8rDeLbpPpgvN/M2P+/nrEkMeAEFZvPf+7wnuIgV9YVJjjxcH1OHUvmmwLUjAzWuoKlaV/XCUjGvNf+2djj8='} 2024-02-14T17:56:49.886 INFO:teuthology.orchestra.console.smithi017:Power off 2024-02-14T17:56:49.886 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-14T17:56:49.915 DEBUG:teuthology.orchestra.console.smithi017:power off output: Chassis Power Control: Down/Off 2024-02-14T17:56:49.916 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:56:49.930 INFO:teuthology.orchestra.console.smithi111:Power off 2024-02-14T17:56:49.931 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-14T17:56:49.943 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-02-14T17:56:49.960 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-02-14T17:56:49.960 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:56:49.996 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-02-14T17:56:53.945 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:56:53.998 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:56:54.058 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is on 2024-02-14T17:56:54.110 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-02-14T17:56:58.062 DEBUG:teuthology.orchestra.console.smithi017:pexpect command: ipmitool -H smithi017.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:56:58.111 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-14T17:56:58.173 DEBUG:teuthology.orchestra.console.smithi017:check power output: Chassis Power is off 2024-02-14T17:56:58.224 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-02-14T17:56:58.274 INFO:teuthology.orchestra.console.smithi017:Power off completed 2024-02-14T17:56:58.326 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-02-14T17:56:58.430 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2853.455049753189 failure_reason: '"2024-02-14T17:15:50.908497+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_gabrioux@teuthology success: false 2024-02-14T17:56:58.430 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-14T17:56:58.525 INFO:teuthology.run:FAIL